UTF BOM

November 23rd, 2007, 8:59 am

I don't even know where to put this post......

W3C warned me I had an UTF BOM in a site I coded, and it might cause problems. So I googled it and I read and I read and I still don't get even what it is, much less how it got there (I sure didn't put it in) nor how to get it out (they give a perl script for it, but how do you use a perl script on a html document? paste it in? or attach it externally like css?) or even whether I should get it out or not.

Briefly my question is this - what are the consequences of an UTF BOM contained in your code?

How does it get there?

How do you get it out?

Thanks in advance..........................

November 23rd, 2007, 12:25 pm

Quote:

they give a perl script for it, but how do you use a perl script on a html document? paste it in? or attach it externally like css?

Chances are you're supposed to pass a directory name when running the script from the command line, then the script will save each file in that directory without the signature/BOM.

What's the script look like ?

November 23rd, 2007, 12:47 pm

Code: [ Download ] [ Select ]

# program to remove a leading UTF-8 BOM from a file
# works both STDIN -> STDOUT and on the spot (with filename as argument)

if ($#ARGV > 0) {
  print STDERR "Too many arguments!\n";
  exit;
  }

my @file;  # file content
my $lineno = 0;

my $filename = @ARGV[0];
if ($filename) {
  open( BOMFILE, $filename ) || die "Could not open source file for reading.";
  while (<BOMFILE>) {
    if ($lineno++ == 0) {
      if ( index( $_, '' ) == 0 ) {
        s/^\xEF\xBB\xBF//;
        print "BOM found and removed.\n";
        }
      else { print "No BOM found.\n"; }
      }
    push @file, $_ ;
    }
  close (BOMFILE) || die "Can't close source file after reading.";

  open (NOBOMFILE, ">$filename") || die "Could not open source file for writing.";
  foreach $line (@file) {
    print NOBOMFILE $line;
    }
  close (NOBOMFILE) || die "Can't close source file after writing.";
  }
else { # STDIN -> STDOUT
  while (<>) {
  if (!$lineno++) {
    s/^\xEF\xBB\xBF//;
    }
  push @file, $_ ;
  }

  foreach $line (@file) {
    print $line;
    }
  }

1. # program to remove a leading UTF-8 BOM from a file
2. # works both STDIN -> STDOUT and on the spot (with filename as argument)
4. if ($#ARGV > 0) {
5.   print STDERR "Too many arguments!\n";
6.   exit;
7.   }
9. my @file;  # file content
10. my $lineno = 0;
12. my $filename = @ARGV[0];
13. if ($filename) {
14.   open( BOMFILE, $filename ) || die "Could not open source file for reading.";
15.   while (<BOMFILE>) {
16.     if ($lineno++ == 0) {
17.       if ( index( $_, '' ) == 0 ) {
18.         s/^\xEF\xBB\xBF//;
19.         print "BOM found and removed.\n";
20.         }
21.       else { print "No BOM found.\n"; }
22.       }
23.     push @file, $_ ;
24.     }
25.   close (BOMFILE) || die "Can't close source file after reading.";
27.   open (NOBOMFILE, ">$filename") || die "Could not open source file for writing.";
28.   foreach $line (@file) {
29.     print NOBOMFILE $line;
30.     }
31.   close (NOBOMFILE) || die "Can't close source file after writing.";
32.   }
33. else { # STDIN -> STDOUT
34.   while (<>) {
35.   if (!$lineno++) {
36.     s/^\xEF\xBB\xBF//;
37.     }
38.   push @file, $_ ;
39.   }
41.   foreach $line (@file) {
42.     print $line;
43.     }
44.   }

November 23rd, 2007, 1:37 pm

It works with single files at once instead of a directory.

On Debian/Linux/etc you would do somthing like this assuming that script is named "debom.pl" & the troubled file is named "bom.html".

Code: [ Download ] [ Select ]

perl /home/me/debom.pl /var/www/bom.html

Windows wouldn't be much different, though Perl is less likely to be installed on a Windows system if you don't already know about it.

Code: [ Download ] [ Select ]

perl "c:\documents and settings\me\desktop\debom.pl" "c:\documents and settings\me\desktop\bom.html

When it's finished that script will print "BOM found and removed." if the file contains a BOM, otherwise it will print "No BOM Found.".

November 25th, 2007, 2:15 am

thanks for that!

December 6th, 2007, 6:45 am

I looked around a bit more on this subject instead of sleeping last nite.

I put together a function in PHP to remove a UTF-8 BOM from a file, but couldn't quite figure out how to detect the BOM at the binary level. I ended up converting the 3 potential BOM bytes into an ASCII hexidecimal & doing a string comparison.

December 10th, 2007, 3:58 am

ok, that's so cool I don't even understand what you said

I'll read the blog entry with great interest....