Saturday, August 6, 2011

Epub Check

‹prev | My Chain | next›

I am still not settled on the exactly how I want to generate mobi formatted versions of SPDY Book. If nothing else today, I would like to ensure that my epub formatted version is solid. To that end, I am going to give epubcheck a whirl.

Sadly, I already have Java installed. So my next step is to download the most recent zip file of epubcheck:
➜  Downloads  wget http://epubcheck.googlecode.com/files/epubcheck-1.2.zip
Google code is pretty cool in that the downloads have SHA1s associated with them, so I do a quick sanity check:
➜  Downloads  sha1sum epubcheck-1.2.zip 
86036eadad8408070791b3da368958239ed8a410  epubcheck-1.2.zip
Yup. Looks good. No unexpected Java code will be running amok on my system.

Next I install:
➜  Downloads  cd ~/src 
➜  src  mkdir epubcheck
➜  src  cd !$
➜  src  cd epubcheck
➜  epubcheck  unzip ~/Downloads/epubcheck-1.2.zip 
Archive:  /home/cstrom/Downloads/epubcheck-1.2.zip
inflating: epubcheck-1.2.jar       
inflating: COPYING.txt             
inflating: README.txt              
inflating: jing_license.txt        
creating: lib/
inflating: lib/jing.jar            
inflating: lib/saxon.jar           
With epubcheck unzipped in $HOME/src, I create a script named epubcheck to run it from $HOME/bin/:
#!/bin/sh

java -jar  $HOME/src/epubcheck/epubcheck-1.2.jar $1
Now, let's see if there are any errors in the epub version of The SPDY Book:
➜  output git:(master) ✗ epubcheck book.epub 
Epubcheck Version 1.2

ERROR: book.epub: resource OEBPS/stylesheets/handbookish.css is missing

Check finished with warnings or errors!
Well, not too bad. What the heck is handbookish.css?

It is nowhere to be found in my local repo for SPDY Book. Maybe it's a part of git-scribe? Hrm... I don't see it there either. Maybe it's generated by git-scribe. Acking through the code, I find reference to it in the test suite, but running the test suite I get two failures on the two lines asserting that handbookish.css should be present:
➜  git-scribe  ruby ./test/gen_test.rb
Loaded suite ./test/gen_test
Started
....FF.
Finished in 12.275742 seconds.

1) Failure:
test_scribe_can_generate_single_page_html(#<Class:0x00000001158000>) [./test/gen_test.rb:23]:
Failed assertion, no message given.

2) Failure:
test_scribe_can_generate_site_html(#<Class:0x00000001158000>) [./test/gen_test.rb:38]:
Failed assertion, no message given.

7 tests, 18 assertions, 2 failures, 0 errors, 0 skips

Test run options: --seed 23414
Commenting out the lines asserting that handbookish.css should be present makes the suite pass. A little investigation and I find that asserting that the scribe.css is present works:
test "scribe can generate site html" do
  in_temp_dir do
    @scribe.init('t')
    Dir.chdir('t') do
      data = @scribe.gen('site')
      out = Dir.glob('output/**/*')
      assert out.include? 'output/index.html'
      assert out.include? 'output/the_first_chapter.html'
      assert out.include? 'output/the_second_chapter.html'
      assert out.include? 'output/image'
        assert out.include? 'output/stylesheets/scribe.css'
end
end
end
I also root through the remainder of the git-scribe source, replacing handbookish.css with scribe.css. After reinstalling the gem, regenerating the epub, now epubcheck works:
➜  output git:(master) ✗ epubcheck book.epub 
Epubcheck Version 1.2

No errors or warnings detected
Ah better. I have put off my mobi toolchain issues for a day, but I feel much better about the epub that I am generating. Now if I could only get an iPad to see how it works in iBooks I might be iSatisfied (so sorry). Maybe tomorrow.


Day #105

Friday, August 5, 2011

Mobi from Epub with Kindlegen

‹prev | My Chain | next›

Last night, I stumbled upon the proper incantation of asciidoc's a2x command to insert cover images into epubs:
a2x -f epub -d book -a docinfo -v book.asc
The key here is the -a docinfo switch, which tells asciidoc to read a file named book-docinfo.xml (or, alternatively, just docinfo.xml). In my case, the docinfo file contains:
<mediaobject role="cover">
<imageobject>
<imagedata fileref="images/cover.jpg" format="JPG"/>
</imageobject>
<textobject><phrase>The SPDY Book</phrase></textobject>
</mediaobject>
This is how I was able to get git-scribe to generate epubs with cover images.

I wonder if I can use that epub to generate nice looking mobi (including the cover image). Let's give it a try with Amazon's kindlegen:
➜  output git:(master) ✗ kindlegen book.epub -o spdybook_from_epub.mobi

**************************************************
* Amazon.com kindlegen(Linux) V1.2 build 33307 *
* A command line e-book compiler *
* Copyright Amazon.com 2011 *
**************************************************

Info(prcgen): Added metadata dc:Title "The SPDY Book"
Info(prcgen): Added metadata dc:Creator "Chris Strom"
Info(prcgen): Parsing files 0000019
Info(cssparser): @rules other than @import and @charset are not supported.
....
Info(prcgen): Resolving hyperlinks
Info(prcgen): Building table of content URL: /tmp/mobi-0K4Jyz/OEBPS/toc.ncx
Info(pagemap): No Page map found in the book
Info(prcgen): Computing UNICODE ranges used in the book
Info(prcgen): Found UNICODE range: Basic Latin [20..7E]
Info(prcgen): Found UNICODE range: General Punctuation - Windows 1252 [2018..201A]
Info(prcgen): Found UNICODE range: Letter-like Symbols [2100..214F]
Info(prcgen): Found UNICODE range: Arrows [2190..21FF]
Info(prcgen): Building MOBI file, record count: 0000064
Info(prcgen): Final stats - text compressed to (in % of original size): 049.01%
Info(prcgen): The document identifier is: "The_SPDY_Book"
Info(prcgen): The file format version is V6
Info(prcgen): Saving MOBI file
Info(prcgen): MOBI File successfully generated!
Hrm... that seems OK. For the most part. I am not sure what the deal with the CSS rules. Similarly, "No Page map"? I wonder what that is about.

First, I check that the output looks OK:



The formatting is still kinda nice. And, no, Scott Chacon is not on the front cover, so there's that.

Now that I think about it, there could be more spacing between paragraphs. The bullet points are on the same line as the associated text (though there may be too much leading whitespace on some bullet points).

A big win is the footnotes, which appear at the end of chapters like normal mobi books (they had been inserted inline).

What is definitely not working is the table of contents. It seems that that "No Page map" warning was in earnest. If I unzip the source epub and check out OEBPS/toc.ncx, it sure seems like there is a page map:
<?xml version="1.0" encoding="utf-8" standalone="no"?>
<ncx:ncx xmlns:ncx="http://www.daisy.org/z3986/2005/ncx/" version="2005-1">
<ncx:head>
<ncx:meta name="cover" content="cover"/>
<ncx:meta name="dtb:depth" content="-1"/>
<ncx:meta name="dtb:totalPageCount" content="0"/>
<ncx:meta name="dtb:maxPageNumber" content="0"/>
</ncx:head>
<ncx:docTitle>
<ncx:text>The SPDY Book</ncx:text>
</ncx:docTitle>
<ncx:navMap>
<ncx:navPoint id="id395944" playOrder="1">
<ncx:navLabel>
<ncx:text>The SPDY Book</ncx:text>
</ncx:navLabel>
<ncx:content src="index.html"/>
<ncx:navPoint id="id395330" playOrder="2">
<ncx:navLabel>
<ncx:text>Copyright</ncx:text>
</ncx:navLabel>
<ncx:content src="pr01.html"/>
</ncx:navPoint>
<ncx:navPoint id="id421493" playOrder="3">
<ncx:navLabel>
<ncx:text>History</ncx:text>
</ncx:navLabel>
<ncx:content src="pr02.html"/>
...
And the chapters themselves seem to contain reasonable headers:
<div class="chapter" title="Chapter 1. A Case For SPDY">
<div class="titlepage">
<div>
<div>
<h1 class="title">
<a id="chapter_a_case_for_spdy"/>
Chapter 1. A Case For SPDY
</h1>
</div>
</div>
</div>
...
Ah, a little research tells me that the page-map thing was just a red-herring. That is apparently for page numbering in the new Kindle software. That's a nice-to-have, but I am not going to worry about that today.

Ugh. So do I work with the existing git-scribe mobi and try to make it more like the epub (e.g. with proper footnotes) or do I work with the mobi generated from the epub and try to get the formatting correct and a TOC?

Dunno that I have a good answer for what's best. I will ruminate on that overnight and pick back up with one of those two approaches tomorrow.


Day #105

Thursday, August 4, 2011

Revisiting ePub and Mobi in Git-Scribe

‹prev | My Chain | next›

Before continuing to pick away at various niggling issues with the mobi spit out by git-scribe, I would like to take some time to investigate alternate approaches to generating the mobi.

Currently, git-scribe uses the asciidoc command to output HTML. It then manually builds wrapper HTML to mark chapters, build a table of contents and perform various other housekeeping work. The asciidoc toolchain is only capable of producing things like PDF, epub and docbook XML, not mobi proper.

The kindlegen command from Amazon, which git-scribe uses to produce mobi, support the following options:
$ kindlegen

**************************************************
* Amazon.com kindlegen(Linux) V1.2 build 33307 *
* A command line e-book compiler *
* Copyright Amazon.com 2011 *
**************************************************

Usage : kindlegen filename.opf/.htm/.html/.epub [-c0 or -c1 or c2] [-verbose] [-rebuild] [-onlydeps or -nodeps] [-western] [-o ]
Options:
-c0: no compression
-c1: standard DOC compression
-c2: Kindle huffdic compression
-o : Specifies the output file name. Output file will be created in the same directory as that of input file. should not contain directory path.
-verbose: verbose output
-rebuild: rebuilds all dependencies
-onlydeps: build only needed dependencies
-nodeps: do not check/build dependencies
-western: force build of Windows-1252 book
-releasenotes: display release notes
-gif: images are converted to GIF format (no JPEG in the book)
So, if I can get a decent epub version of SPDY Book, maybe I can generate a nicer mobi from that. Unfortunately, the epub produced by git-scribe looks like this (in calibre):



For SPDY Book, I found that the best PDF was produced by generating docbook and then running that through xsl-fo. I did not do any such tweaking for epub. Unfortunately, the xsl-fo stuff does not generate epub either. So how am I to get a cover and other useful stuff in there?

Ooh! The asciidoc web site actually has a relatively nice example of doing something like that. If I follow correctly, something along the lines of the following ought to work:
a2x -f epub -d book -a docinfo book.asc -v
With the following used as the docbook manifest (book-docinfo.xml):
<mediaobject role="cover">
<imageobject>
<imagedata fileref="images/cover.jpg" format="JPG"/>
</imageobject>
<textobject><phrase>The SPDY Book</phrase></textobject>
</mediaobject>
And indeed, that seems to work. The cover image is showing up along with my book in calibre:



Even better, the table of contents and formatting seems pretty nice:



I cannot believe that it actually kinda sorta works. I will go back tomorrow to start incorporating some of this in git-scribe. And I still need to figure out how well this translates into mobi.


Day #104

Wednesday, August 3, 2011

Fixed Width Font with Git-scribe

‹prev | My Chain | next›

Of my three immediate problems with git-scribe's mobi formatting, I have solved all but the code formatting. I have the book cover on the mobi front page. I have the bullet points on the same line with the bulleted text. Now I would like to get the code a little smaller so that it fits better on a Kindle.

I really hope this is a quick fix.

And it turns out to be. In stylesheets/scribe.css, I change the fixed font size to 8px:
tt {
font-family:consolas, 'lucida console', 'bitstream vera sans mono',
'courier new', monospace;
color:#000;
font-size: 8px;
}
That does make things smaller. But not quite small enough.

I had targeted 50 characters width in SPDY Book for all of my code samples. I generally aim for 60 characters in real world code, so 50 characters did not seem that far-fetched. Unfortunately, any code sample in SPDY Book that were just about 50 characters ended up wrapping. Dammit.

So I try 4px and even 2px for the font-size of tt text. To no avail. It seems that there is a hard, minimum font-size for fixed width text in the Kindle because the font-size remained the same—even when set to 2px.

Ah well. Maybe for the second edition of SPDY Book I will rework the code samples to be 45 characters or less. For now, I think I can live with a few lines wrapping.



Day #102

Tuesday, August 2, 2011

Better Git-scribe Bullet Lists on Kindle

‹prev | My Chain | next›

Today, I continue my efforts to improve the mobi formatted version of SPDY Book (which you should totally buy, it's awesome). One of the problems in the mobi is that, on the kinde, bullet lists are all out of whack.

On the mobi readers that I have on Ubuntu, the bullet lists look fine. On the kindle, however, the text starts on the line below the bullet. Inspecting the element in Calibre, I see:



Some of that is certainly added by Calibre. Still, I suspect that the problem on the kindle is a <p> tag inside a <li> tag. Hopefully I can remove it. If nothing else, I will run it through sed. To the git-scribe source!

In generate.rb, the do_mobi method reads:
    def do_mobi
do_html
info "GENERATING MOBI"
generate_toc_files
# generate book.opf
cmd = "kindlegen -verbose book.opf -o book.mobi"
if ex(cmd)
'book.mobi'
end
end
The first place I will look, then, is that book.opf that served as the input to kindlegen. If it has <p> tags inside a <li> tags, then I can concentrate my efforts there. If not, I will have to investigate kindlegen itself.

Hrm... book.opf is fairly empty. It has a top-level <package> tag that contains four children: <metadata>, <manifest>, <spine>, and <guide>. Both <manifest> and <guide> reference book.html, so I check there next.

The book.html file is pretty cool. It not only has the bullet list that is causing me problems, but includes CSS and JS(!) as well. I will be keen to see the Javascript in action at some point (it seems to do footnotes somehow) and I definitely need to fiddle with the CSS. But, for now, I am only interested in the lists, which look like:
<div class="ulist"><ul>
<li>
<p>
2011-06-30: First beta release.
</p>
</li>
<li>
<p>
2011-07-05: Formatting changes. Copy edits from Ashish Dixit.
</p>
</li>
Yup. Stinking <p> tags. To verify that this is the source of my bullet list woes, I manually remove the <p> tags, manually regenerate the mobi (using the same kindlegen command from generate.rb), and copy the file to my kindle:
➜  output git:(master) ✗ kindlegen -verbose book.opf -o book.mobi
➜ output git:(master) ✗ cp book.mobi /media/Kindle/documents/spdybook.mobi
And that fixes it! But I am not going to manually edit book.html each time I edit SPDY Book.

Before trying to alter the output of asciidoc, I try a pure CSS solution. Git-scribe uses the following to style <p> and <li> tags:
p, ul, ol, dl {
margin:10px 0
}
I get nowhere with that, however. Making p display inline, removing the margin, special styles for p tags inside li tags all have no effect.

So I resort to manually removing the extra p tag:
    def do_html
return true if @done['html']
info "GENERATING HTML"
# TODO: look for custom stylesheets
styledir = local('stylesheets')
cmd = "asciidoc -a stylesdir=#{styledir} -a theme=scribe #{BOOK_FILE}"
if ex(cmd)
remove_p_from_li('book.html')
@done['html'] == true
'book.html'
end
end
My sed and awk-fu are not powerful enough to work on multi-line text, so I have to resort to slurping the entire book into a buffer and gsub'ing on it:
    def remove_p_from_li(file)
content = File.read(file)
File.open(file, 'w') do |f|
f.write content.gsub(%r"<li>\s*<p>(.+?)</p>\s*</li>"m, '<li>\1</li>')
end
end
There is simply no way that is going to work for all books. But that will do for now as it successfully removes the p tags from my output:
<h2 id="_history">History</h2>
<div class="sectionbody">
<div class="ulist"><ul>
<li>
2011-06-30: First beta release.
</li>
<li>
2011-07-05: Formatting changes. Copy edits from Ashish Dixit.
</li>
The resulting mobi looks good on the kindle so I check that off my TODO list. But add another, future TODO item to come up with a more robust solution.


Day #101

Monday, August 1, 2011

Arrgh! Why is Scott Chacon Still on the Cover of SPDY Book, Kindle Edition?

‹prev | My Chain | next›

Somehow Scott Chacon's picture is on the front of the Kindle (mobi) edition of SPDY Book. I am using his very nice git-scribe toolchain to maintain and generate SPDY Book, so that is at least a partial explanation. Still, I swear that I have deleted his picture from every possible place in my source directory, but it keeps popping back up. He tasks me. He tasks me. And I shall have him. Gone. Dammit.



I start by searching for cover images:
➜  spdybook git:(master) find -ls| grep cover
2280514 384 -rw-r--r-- 1 cstrom cstrom 389941 Jul 31 23:56 ./output/images/cover.png
2279528 60 -rw-r--r-- 1 cstrom cstrom 59454 Jul 21 21:17 ./output/image/cover.jpg
2280317 384 -rw-r--r-- 1 cstrom cstrom 389941 Jul 25 01:07 ./book/images/cover.png
Dang it! I had deleted that cover.jpg from the source directory. I neglected to get rid of it from the generated output directory. It's been happily finding its way into the mobi formatted book since July 21. Ah well, a quick rm ./output/image/cover.jpg and he is once and forever gone:



Of course, it would be nice to get my real cover in the mobi version. I have the cover in PNG format, but it is easy enough to convert it to the JPEG required by git-scribe:
➜  image git:(master) ✗ convert -format jpg ../images/cover.png cover.jpg
➜ image git:(master) ✗ ls -l
total 120
-rw-r--r-- 1 cstrom cstrom 119604 2011-08-01 22:22 cover.jpg
With that, I have my sweet SPDY cover back:



I would prefer the image directory to be plural. For SPDY Book, I put all of my images in the chapter directories. Even so, a book can have multiple cover images (front, back, inside, etc). The default cover location is specified in the lib/git-scribe/generate.rb:
      File.open('book.opf', 'w+') do |f|
lang = @config['language'] || 'en'
author = @config['author'] || 'Author'
cover = @config['cover'] || 'image/cover.jpg'
data = {'title' => book_title,
'language' => lang,
'author' => author,
'pubdate' => Time.now.strftime("%Y-%m-%d"),
'cover_image' => cover}
f.puts opf_template.render( data )
end
So I change the location:
        cover  = @config['cover'] || 'images/cover.jpg'
I also modify my local changes to the docbook XSL-FO so that the PDF will include cover.jpg instead of the PNG version. I re-install my local gem:
➜  git-scribe git:(master) gem build git-scribe.gemspec; gem install git-scribe-0.0.9.gem
...
Successfully built RubyGem
Name: git-scribe
Version: 0.0.9
File: git-scribe-0.0.9.gem
Successfully installed git-scribe-0.0.9
1 gem installed
Then I remove the PNG cover image and the image directory (from both book and output). With that, I re-generate my mobi and PDF copies of SPDY Book. Now they both have the right cover. And no trace of Scott Chacon!

Day #100

Never Ending Chain

‹prev | My Chain | next›

My SPDY chain may be complete, but that doesn't mean it is time to break my chain. I'm just getting started here!

For the time being, I am going to switch to improving my ebook toolchain (and SPDY Book touch-up as needed). I used git-scribe to produce SPDY Book. It's a pretty good tool, but the resultant output is a bit... rough.

I am going to go back and redo some of my awful hacks. I am also going to get mobi formatting working. It's pretty terrible. And it puts Scott Chacon's picture on the first page of my book. Don't get me wrong. He's a handsome man and all, but it's my damn book!

Once I'm satisfied with my toolchain? You guessed it. I'm onto my next book. Got any suggestions for topics?


Day #100 / #1