File Coverage

=head1 NAME

C4::Charset - utilities for handling character set conversions.

=head1 SYNOPSIS

  use C4::Charset;

=head1 DESCRIPTION

This module contains routines for dealing with character set
conversions, particularly for MARC records.

A variety of character encodings are in use by various MARC
standards, and even more character encodings are used by
non-standard MARC records.  The various MARC formats generally
do not do a good job of advertising a given record's character
encoding, and even when a record does advertise its encoding,
e.g., via the Leader/09, experience has shown that one cannot
trust it.

Ultimately, all MARC records are stored in Koha in UTF-8 and
must be converted from whatever the source character encoding is.
The goal of this module is to ensure that these conversions
take place accurately.  When a character conversion cannot take
place, or at least not accurately, the module was provide
enough information to allow user-facing code to inform the user
on how to deal with the situation.

=cut

=head1 FUNCTIONS

=head2 IsStringUTF8ish

  my $is_utf8 = IsStringUTF8ish($str);

Determines if C<$str> is valid UTF-8.  This can mean
one of two things:

=over

=item *

The Perl UTF-8 flag is set and the string contains valid UTF-8.

=item *

The Perl UTF-8 flag is B<not> set, but the octets contain
valid UTF-8.

=back

The function is named C<IsStringUTF8ish> instead of C<IsStringUTF8> 
because in one could be presented with a MARC blob that is
not actually in UTF-8 but whose sequence of octets appears to be
valid UTF-8.  The rest of the MARC character conversion functions 
will assume that this situation occur does not very often.

=cut

=head2 SetUTF8Flag

  my $marc_record = SetUTF8Flag($marc_record, $nfd);

This function sets the PERL UTF8 flag for data.
It is required when using new_from_usmarc 
since MARC::File::USMARC does not handle PERL UTF8 setting.
When editing unicode marc records fields and subfields, you
would end up in double encoding without using this function. 

If $nfd is set, string normalization will use NFD instead of NFC

FIXME
In my opinion, this function belongs to MARC::Record and not
to this package.
But since it handles charset, and MARC::Record, it finds its way in that package

=cut

=head2 NormalizeString

    my $normalized_string=NormalizeString($string,$nfd,$transform);

Given a string
nfd : If you want to set NFD and not NFC
transform : If you expect all the signs to be removed

Sets the PERL UTF8 Flag on your initial data if need be
and applies cleaning if required

Returns a utf8 NFC normalized string

Sample code :
   my $string=NormalizeString ("l'ornithoptère");
   #results into ornithoptère in NFC form and sets UTF8 Flag

=cut

=head2 MarcToUTF8Record

  ($marc_record, $converted_from, $errors_arrayref) = MarcToUTF8Record($marc_blob, 
					$marc_flavour, [, $source_encoding]);

Given a MARC blob or a C<MARC::Record>, the MARC flavour, and an 
optional source encoding, return a C<MARC::Record> that is 
converted to UTF-8.

The returned C<$marc_record> is guaranteed to be in valid UTF-8, but
is not guaranteed to have been converted correctly.  Specifically,
if C<$converted_from> is 'failed', the MARC record returned failed
character conversion and had each of its non-ASCII octets changed
to the Unicode replacement character.

If the source encoding was not specified, this routine will 
try to guess it; the character encoding used for a successful
conversion is returned in C<$converted_from>.

=cut

=head2 SetMarcUnicodeFlag

  SetMarcUnicodeFlag($marc_record, $marc_flavour);

Set both the internal MARC::Record encoding flag
and the appropriate Leader/09 (MARC21) or 
100/26-29 (UNIMARC) to indicate that the record
is in UTF-8.  Note that this does B<not> do
any actual character conversion.

=cut

=head2 StripNonXmlChars

  my $new_str = StripNonXmlChars($old_str);

Given a string, return a copy with the
characters that are illegal in XML 
removed.

This function exists to work around a problem
that can occur with badly-encoded MARC records.
Specifically, if a UTF-8 MARC record also
has excape (\x1b) characters, MARC::File::XML
will let the escape characters pass through
when as_xml() or as_xml_record() is called.  The
problem is that the escape character is not
legal in well-formed XML documents, so when
MARC::File::XML attempts to parse such a record,
the XML parser will fail.

Stripping such characters will allow a 
MARC::Record->new_from_xml()
to work, at the possible risk of some data loss.

=cut

=head2 nsb_clean

=over 4

nsb_clean($string);

=back

Removes Non Sorting Block characters

=cut

=head1 INTERNAL FUNCTIONS

=head2 _default_marc21_charconv_to_utf8

  my ($new_marc_record, $guessed_charset) = _default_marc21_charconv_to_utf8($marc_record);

Converts a C<MARC::Record> of unknown character set to UTF-8,
first by trying a MARC-8 to UTF-8 conversion, then ISO-8859-1
to UTF-8, then a default conversion that replaces each non-ASCII
character with the replacement character.

The C<$guessed_charset> return value contains the character set
that resulted in a conversion to valid UTF-8; note that
if the MARC-8 and ISO-8859-1 conversions failed, the value of
this is 'failed'. 

=cut

=head2 _default_unimarc_charconv_to_utf8

  my ($new_marc_record, $guessed_charset) = _default_unimarc_charconv_to_utf8($marc_record);

Converts a C<MARC::Record> of unknown character set to UTF-8,
first by trying a ISO-5426 to UTF-8 conversion, then ISO-8859-1
to UTF-8, then a default conversion that replaces each non-ASCII
character with the replacement character.

The C<$guessed_charset> return value contains the character set
that resulted in a conversion to valid UTF-8; note that
if the MARC-8 and ISO-8859-1 conversions failed, the value of
this is 'failed'. 

=cut

=head2 _marc_marc8_to_utf8

  my @errors = _marc_marc8_to_utf8($marc_record, $marc_flavour, $source_encoding);

Convert a C<MARC::Record> to UTF-8 in-place from MARC-8.
If the conversion fails for some reason, an
appropriate messages will be placed in the returned
C<@errors> array.

=cut

=head2 _marc_iso5426_to_utf8

  my @errors = _marc_iso5426_to_utf8($marc_record, $marc_flavour, $source_encoding);

Convert a C<MARC::Record> to UTF-8 in-place from ISO-5426.
If the conversion fails for some reason, an
appropriate messages will be placed in the returned
C<@errors> array.

FIXME - is ISO-5426 equivalent enough to MARC-8
that C<MARC::Charset> can be used instead?

=cut

=head2 _marc_to_utf8_via_text_iconv 

  my @errors = _marc_to_utf8_via_text_iconv($marc_record, $marc_flavour, $source_encoding);

Convert a C<MARC::Record> to UTF-8 in-place using the
C<Text::Iconv> CPAN module.  Any source encoding accepted
by the user's iconv installation should work.  If
the source encoding is not recognized on the user's 
server or the conversion fails for some reason,
appropriate messages will be placed in the returned
C<@errors> array.

=cut

=head2 _marc_to_utf8_replacement_char 

  _marc_to_utf8_replacement_char($marc_record, $marc_flavour);

Convert a C<MARC::Record> to UTF-8 in-place, adopting the 
unsatisfactory method of replacing all non-ASCII (e.g.,
where the eight bit is set) octet with the Unicode
replacement character.  This is meant as a last-ditch
method, and would be best used as part of a UI that
lets a cataloguer pick various character conversions
until he or she finds the right one.

=cut

=head2 char_decode5426

  my $utf8string = char_decode5426($iso_5426_string);

Converts a string from ISO-5426 to UTF-8.

=cut

=head1 AUTHOR

Koha Development Team <http://koha-community.org/>

Galen Charlton <galen.charlton@liblime.com>

=cut