Git-Scribe Rebase (Interactive)

‹prev | My Chain | next›

I more or less successfully rebased my fork of git-scribe onto upstream. There are still a few things that I need to check out, but I am left with 39 post-upstream commits. Some of them should probably be merged together, so I set off...

➜  git-scribe git:(master) git rebase -i upstream/master

Some of those commits are easy to squash together:

...
pick 24f4048 Grey background for code samples in epub
pick 1e28f14 Note on my fork
s c5c1cae Note on my fork
s f85b11c Note on my fork
s 63e6716 Note on my fork
pick a41d779 Support both admonition icons and cover image.
...

Others... not so much.

I am not sure I'd recommend this, but it is possible to re-order commits during an interactive rebase. The README was updated twice:

commit 84e5f4e45bfdaf7f93bec4a1f6922421a1c6f620
Author: Chris Strom 
Date:   Sun Apr 8 15:39:44 2012 -0400

    Note on my fork

commit 95c96f2501c4467fdd2a5a757db935a7619be60c
Author: Chris Strom 
Date:   Sun Aug 14 21:32:32 2011 -0400

    Updated README

But those commits are far apart in the history:

...
* 3f11995 Support both admonition icons and cover image.
* 84e5f4e Note on my fork
* c2d687b Grey background for code samples in epub
* 052d2e4 Admonition icons for PDF and epub.
....
* 4b89856 The generators only return boolean on completion.
* bf1bc34 Fix site html test.
* 95c96f2 Updated README
* 6d226fa Prepare the site in a separate sub-directory.
...

During yet another git rebase -i upstream/master, I can make those the last two commits, and squash them together:

...
pick 96e3541 Test tweak for source highlights
pick 95c96f2 Updated README
s 84e5f4e Note on my fork

In the end, I get it down to 24 commits between upstream and HEAD on my fork. The changes break down as:

• Note about my fork
• Test fixes (3)
• Admonition icons
• Epub: grey background for epub
• Minor fix: useless puts
• Post-commit hook for mobi (I use for ebok)
• TODO notes for PDF
• Same fix, different place (1)
• Renames image to images
• Build site separately to avoid conflict
• Support ZIP of three formats (git-scribe ebook)
• Remove temporary work files
• Code re-org -- I strongly prefer defining methods after their first use (6)
• Better mobi from epub (cover image, TOC) (2)
• Cover image in PDF
• PDF should be 7.5"x9"

I made the unfortunate choice to re-organize much of the generator code early on in my fork. It helped me out greatly when I was adding some of the features, but it leaves me in a tough position for tracking upstream. Compounding that is that the very first change in my fork—the 7.5" x 9" PDF form factor—conflicts with later upstream changes that switch XSLT processors.

Of the changes, I can definitely send a pull request to get rid of the useless puts statement. That was an annoyance every time I generated an updated gem. But aside from that, there is not much I can push in. Even the admonition icons do not apply well—it almost works until I realize that it relies on building up a docinfo file that I did last year to get the image cover in the epub. Bah.

I think that I am forced to call it a day here. I have addressed a few bugs in my fork as well as upstream. I have fixed various tests in both forks as well. I have improved the PDF and epub formats in my fork. Finally, I have rebased my fork back onto upstream so I have a better idea of what I have added and why. In the end, I want the features that I have added, so I am unlikely to strip things down to make a complete pull request possible. At the same time, I understand why upstream made the changes it made.

I have heard rumor that git-scribe is about to undergo significant change. That is probably for the best—especially if it pulls in the features of both forks without the baggage of either approach. /me waits eagerly.

In the meantime, it is time to get back to SPDY.


Day #351

Git-Scribe Rebase

‹prev | My Chain | next›

I am sorely tempted to move along with SPDY Book today, but how can I with my fork having so badly diverged from git-scribe? A git-rebase is gonna be painful, but what kind of citizen of the OSS world am I if I do not at least try?

And, indeed, it is painful. I lose count of the number of conflicts along the way, but I eventually reach the last conflict and fix it:

➜  git-scribe(master)  git fetch upstream  
➜  git-scribe(master)  git rebase upstream/master
...
Applying: Re-org XSL cover page "code" in clean code order
Using index info to reconstruct a base tree...
Falling back to patching base and 3-way merge...
Auto-merging docbook-xsl/fo.xsl
CONFLICT (content): Merge conflict in docbook-xsl/fo.xsl
Failed to merge in the changes.
Patch failed at 0030 Re-org XSL cover page "code" in clean code order

When you have resolved this problem run "git rebase --continue".
If you would prefer to skip this patch, instead run "git rebase --skip".
To restore the original branch and stop rebasing run "git rebase --abort".

➜  git-scribe  gst
# Not currently on any branch.
# Unmerged paths:
#   (use "git reset HEAD ..." to unstage)
#   (use "git add/rm ..." as appropriate to mark resolution)
#
#       both modified:      docbook-xsl/fo.xsl
#
no changes added to commit (use "git add" and/or "git commit -a")
➜  git-scribe  git add docbook-xsl/fo.xsl
➜  git-scribe  git rebase --continue     
Applying: Re-org XSL cover page "code" in clean code order
Applying: ensure that code blocks don't break over pages, making it way easier to read
Using index info to reconstruct a base tree...
Falling back to patching base and 3-way merge...
No changes -- Patch already applied.
Applying: Post commit-hook for mobi processing.
Applying: Remove useless puts
Applying: Remove copy dest to deal with symbolic links
Applying: Admonition icons for PDF and epub.
Applying: Grey background for code samples in epub
Applying: Note on my fork

I keep a reference to my original fork so that I can tell what might have changed after rebasing. My original fork was working pretty well for me and I do not want to suffer for having done the right thing. The reference is at:

➜  git-scribe git:(master) git co -b original-fork b80326e
Switched to a new branch 'original-fork'

Testing the output in a bunch of different devices is a huge, long feedback cycle pain, so first up, I run the test suite:

➜  git-scribe git:(master) rake
Started
...F.F....E.FFF...

1) Failure:
test_scribe_can_generate_a_mobi(scribe_gen_tests)

2) Failure:
test_scribe_can_generate_a_pdf_with_syntax_highlighting(scribe_gen_tests)

3) Error:
test_scribe_can_generate_site_with_syntax_highlighting(scribe_gen_tests):

4) Failure:
test_scribe_generates_a_mobi_with_a_cover_and_TOC(scribe_gen_tests)

5) Failure:
test_scribe_generates_an_epub_with_a_cover(scribe_gen_tests)

6) Failure:
test_will_not_respond_to_non_thing(scribe_gen_tests) [/home/cstrom/repos/git-scribe/test/gen_test.rb:9]:

18 tests, 38 assertions, 5 failures, 1 errors, 0 pendings, 0 omissions, 0 notifications
66.6667% passed

0.41 tests/s, 0.86 assertions/s

Ugh. The syntax highlighting errors I expect—that just will not work on Ubuntu since those tests come from upstream, which is pegged to OSX. The cover errors are more worrisome. They turn out to have been introduced by my lovely admonition icons, which I added to my docinfo template (used by the underlying asciidoc tool chain). I added them to the cover media object, which seems to wreak havoc with the actual cover image:

<mediaobject role="cover">
  <imageobject>
    <imagedata fileref="images/icons/note.png" format="PNG"/>
  </imageobject>
  <imageobject>
    <imagedata fileref="images/icons/caution.png" format="PNG"/>
  </imageobject>
  <imageobject>
    <imagedata fileref="images/icons/important.png" format="PNG"/>
  </imageobject>
  <imageobject>
    <imagedata fileref="images/icons/tip.png" format="PNG"/>
  </imageobject>
  <imageobject>
    <imagedata fileref="images/icons/warning.png" format="PNG"/>
  </imageobject>
  <imageobject>
    <imagedata fileref="{{cover_image}}" format="JPG"/>
  </imageobject>
  <textobject><phrase>{{title}}</phrase></textobject>
</mediaobject>

Removing those images gets the tests passing, but now I lack for my lovely admonition icons.

I am forced to call it a day here. Hopefully tomorrow I can find another place to include these icons so that nothing breaks.

Update: Figured out a way to do this. I move the admonition icons into a separate mediaobject:

<mediaobject role="cover">
  <imageobject>
    <imagedata fileref="{{cover_image}}" format="JPG"/>
  </imageobject>
  <textobject><phrase>{{title}}</phrase></textobject>
</mediaobject>
<mediaobject role="icons">
  <imageobject>
    <imagedata fileref="images/icons/note.png" format="PNG"/>
  </imageobject>
  <imageobject>
    <imagedata fileref="images/icons/caution.png" format="PNG"/>
  </imageobject>
  <imageobject>
    <imagedata fileref="images/icons/important.png" format="PNG"/>
  </imageobject>
  <imageobject>
    <imagedata fileref="images/icons/tip.png" format="PNG"/>
  </imageobject>
  <imageobject>
    <imagedata fileref="images/icons/warning.png" format="PNG"/>
  </imageobject>
</mediaobject>

That seems to be enough for the underlying asciidoc / a2x toolchain to pick up both the cover image and the admonition icons. Multiple root elements in an XML file like book-docinfo.xml does not seem like a recipe for long term success. But I'll take a short term win. Especially when there are unit tests to catch regressions.


Day #350

Tweaking Code Samples in epub

‹prev | My Chain | next›

While mucking with the epub version of Dart for Hipsters, I have noticed that the code formatting leaves a little something to be desired. Sure, I'd like to have syntax highlighting working, but that's a battle for another book, I think.

I track down the CSS to stylesheets/scribe.css in git-scribe. To that, I add a light grey background:

pre, tt {
 font-family:consolas, 'lucida console', 'bitstream vera sans mono',
            'courier new', monospace;
 color: #000;
  background-color: #eee;
}

I am copying the epubs over to the ipad via dropbox, but the subsequent imports into iBook seemingly make no difference. The result when viewed in Calibre includes the correct background. I eventually get this into the iPad by naming the files differently with each copy to dropbox (e.g. book01.epub, book02.epub, etc). I am not quite certain why this was needed or if it was needed each time, but this is one of the joys of working with book formats—the long feedback cycle when developing.

At any rate, the result is something like:

Which is an improvement. I experiment some with different font-weights, but bold feels too heavy:

And font weights like 500 have no effect (just like anywhere else). So I settle for the light grey and call it a night.


Day #349

Admonitions in Git-Scribe

‹prev | My Chain | next›

None of my books include admonition icons, which are the little icons next to NOTE / WARNING / IMPORTANT sections. I have never really given this too much thought, but this seems like it might be a nice touch. If nothing else, it might help to make these sections more obvious.

In the git-scribe source these are explicitly disabled:

    def do_pdf
      # ...
      strparams = {'callout.graphics' => 1,
                   'navig.graphics' => 1,
                   'admon.textlabel' => 1,
                   'admon.graphics' => 0,
                   'page.width' => '7.5in',
                   'page.height' => '9in'}
      # ...
    end

It seems a simple enough matter to turn them on with 'admon.graphics' => 1, but first, I will try a simple, command-line test case to verify that this works at all.

The other day, I figured out that the following will generate HTML for a single chapter in Dart for Hipsters:

a2x -f xhtml -v dom.asc

The admon.graphics option looks to be off by default because the result of that command is icon-less NOTEs:

The admin.graphics option needs to be sent to the xsltproc sub-command via --xsltproc-opts:

a2x -f xhtml -v --xsltproc-opts='--stringparam admon.graphics 1' dom.asc

With that, I have my icons:

So, I ought to be ready to try the option in git-scribe:

    def do_pdf
      # ...
      strparams = {'callout.graphics' => 1,
                   'navig.graphics' => 1,
                   'admon.textlabel' => 1,
                   'admon.graphics' => 1,
                   'page.width' => '7.5in',
                   'page.height' => '9in'}
      # ...
    end

But, when I run git-scribe gen pdf, I get lots of icon-not-found errors:

...
[ERROR] FONode - Image not found: images/icons/note.png
[ERROR] render - Image not found: images/icons/note.png
[ERROR] FONode - Image not found: images/icons/important.png
[ERROR] FONode - Image not found: images/icons/note.png
...

And, indeed, the FO document is refering to that path:

<fo:block>
  <fo:external-graphic
     width="auto"
     height="auto"
     content-width="36pt"
     src="url(images/icons/note.png)"/>
</fo:block>

To fix, I copy the image icons into the output directory along with the stylesheets and book content that is already copied when prepping the output directory:

    def prepare_output_dir(dir='output')
      FileUtils.mkdir_p(dir)
      FileUtils.cp_r "#{@wd}/book/.", dir, :remove_destination => true

      FileUtils.mkdir_p("#{dir}/stylesheets")
      FileUtils.cp_r File.join(SCRIBE_ROOT, 'stylesheets'), dir

      FileUtils.mkdir_p("#{dir}/images/icons")
      FileUtils.cp_r File.join(SCRIBE_ROOT, 'docbook-xsl/images/.'),
                     "#{dir}/images/icons"
    end

Now, when I git-scribe gen pdf, I get my icons:

Well, not quite my icons. It seems that the docbook icons included with git-scribe are not nearly as pretty as the AsciiDoc icons. So, I copy the AsciiDoc icons included in my normal Ubuntu install into git-scribe:

➜  git-scribe git:(master) ✗ mkdir icons
➜  git-scribe git:(master) ✗ cp -r /usr/share/xml/docbook/stylesheet/docbook-xsl/images/. icons 

Then I can update prepare_output_dir to copy these icons instead of the ones bundled in the docbook:

    def prepare_output_dir(dir='output')
      FileUtils.mkdir_p(dir)
      FileUtils.cp_r "#{@wd}/book/.", dir, :remove_destination => true

      FileUtils.mkdir_p("#{dir}/stylesheets")
      FileUtils.cp_r File.join(SCRIBE_ROOT, 'stylesheets'), dir

      FileUtils.mkdir_p("#{dir}/images/icons")
      FileUtils.cp_r File.join(SCRIBE_ROOT, 'icons'),
                     "#{dir}/images"
    end

With that, I have pretty icons in my PDF output:

I also add these images to the epub (and maybe the mobi) before calling it a night. Up tomorrow? Who knows?


Day #348

Git-Scribe Fixes

‹prev | My Chain | next›

My fork and git-scribe have diverged significantly of late. I am unsure how much effort it will take to get the two back in sync, but there is only one way to find out.

Before getting into a rebase too deep, I think I will dip my toes in with a small cherry pick. The no-clobber symlinks work from the other day seems like a good place to start.

I added the real git-scribe as an upstream remote a while back:

➜  git-scribe git:(master) git co -b upstream upstream/master
Branch upstream set up to track remote branch master from upstream.
Switched to a new branch 'upstream'

So, back in that branch, I bundle install and try running the tests:

➜  git-scribe git:(upstream) rake
/home/cstrom/.rvm/rubies/ruby-1.9.3-p0-falcon/bin/ruby -I"lib:test" -I"/home/cstrom/.rvm/gems/ruby-1.9.3-p0-falcon@global/gems/rake-0.9.2.2/lib" "/home/cstrom/.rvm/gems/ruby-1.9.3-
p0-falcon@global/gems/rake-0.9.2.2/lib/rake/rake_test_loader.rb" "test/check_test.rb" "test/init_test.rb" "test/gen_test.rb"
Loaded suite /home/cstrom/.rvm/gems/ruby-1.9.3-p0-falcon@global/gems/rake-0.9.2.2/lib/rake/rake_test_loader
Started
...FMaking portrait pages on A4 paper (210mmx297mm)
.Making portrait pages on A4 paper (210mmx297mm)
..FXSLT Highlighter: Cannot read xslthl-config.xml, no custom highlighters will be available.
Writing ar01s02.html for section(_the_first_chapter)
Writing ar01s03.html for section(_the_second_chapter)
Writing index.html for article
FXSLT Highlighter: Cannot read xslthl-config.xml, no custom highlighters will be available.
Writing ar01s02.html for section(_the_first_chapter)
Writing ar01s03.html for section(_the_second_chapter)
Writing index.html for article
F....

1) Failure:
test_scribe_can_generate_a_mobi(scribe_gen_tests)
  ...
<nil> expected but was
<"book.mobi">
diff:
? n        il
? "book.mob "

2) Failure:
test_scribe_can_generate_single_page_html(scribe_gen_tests)
  ...
lt;top (required)>']:
<false> is not true.

3) Failure:
test_scribe_can_generate_site_html(scribe_gen_tests)
  ...
<false> is not true.

4) Failure:
test_scribe_can_generate_site_with_syntax_highlighting(scribe_gen_tests)
  ...
<nil> is not true.

Finished in 137.003675962 seconds.

Ugh. I know that some of those failures are because the upstream branch has been pegged to OSX Saxon configuration. I got that more or less working late last year under Ubuntu. Many of those changes consisted in editing XML files, so it is an open question how to get that working cross platform.

Also, 137 seconds?!

Anyhow, let's see if I can fix those tests (aside from the Saxon failures). I track one down to the simple fact that the images used to be named "image". Hrm... that seems familiar:

➜  git-scribe git:(upstream) ✗ find | grep cover.jpg
./template/book/images/cover.jpg
➜  git-scribe git:(upstream) ✗ git log ./template/book/images/cover.jpg
commit 2cce41f371392d7b78de45ad635efec2ab6110b6
Author: Chris Strom <eee.c@eeecooks.com>
Date:   Mon Aug 1 23:13:59 2011 -0400

    Books can have multiple images.
    
    Back cover, inside cover, etc.  Pluralize template image directory to accommodate.

Dammit. I didn't realize that any of my pull requests had made it in there. And without tests? Tsk.

The fix for two of the failures is simple enough—change image to images. The mobi test failure is a little trickier. In addition to changing an image include to images, I keep getting warnings about the cover image from the kindlegen sub-process:

...
Warning(prcgen): Cover file not found :   /tmp/test20120405-441-1lcib49/t/output/image/cover.jpg
Warning(prcgen): media file not found  /tmp/test20120405-441-1lcib49/t/output/image/cover.jpg
Warning(prcgen): media file not found  /tmp/test20120405-441-1lcib49/t/output/image/cover.jpg
...

That is not specified in the test file, rather it is in the OPF generation:

      File.open('book.opf', 'w+') do |f|
        lang   = @config['language'] || 'en'
        author = @config['author'] || 'Author'
        cover  = @config['cover'] || 'image/cover.jpg'
        ...

OK so maybe not that tricky.

After fixing that by adding an "s", I am ready to try adding my own test / patch. The test case turns out to be more involved than an error when re-copying symlinks with FileUtils.cp_r. Rather, I can only make the test fail when the symbolic link and target file are in separate directories:

  test "scribe don't crash on symlinks when run twice" do
    in_temp_dir do
      @scribe.init('t')
      Dir.chdir('t') do
        FileUtils.mkdir_p 'book/includes/sub1'
        FileUtils.mkdir_p 'book/includes/sub2'
        FileUtils.touch 'book/includes/sub1/real_file'
        FileUtils.ln_s '../sub1/real_file', 'book/includes/sub2/link_file'

        assert_nothing_raised do
          @scribe.gen('html')
          @scribe.gen('html')
        end
      end
    end
  end

With that, I get a failure from the second @scribe.gen():

➜  git-scribe git:(test-fixes) ✗ ruby test/gen_test.rb -n test_scribe_don_t_crash_on_symlinks_when_run_twice
Loaded suite test/gen_test
Started
F

1) Failure:
test_scribe_don_t_crash_on_symlinks_when_run_twice(scribe_gen_tests)
    [test/gen_test.rb:23:in `block (4 levels) in <main>'
     test/gen_test.rb:17:in `chdir'
     test/gen_test.rb:17:in `block (3 levels) in <main>'
     /home/cstrom/repos/git-scribe/test/test_helper.rb:40:in `block in in_temp_dir'
     /home/cstrom/repos/git-scribe/test/test_helper.rb:39:in `chdir'
     /home/cstrom/repos/git-scribe/test/test_helper.rb:39:in `in_temp_dir'
     test/gen_test.rb:15:in `block (2 levels) in <main>']:
Exception raised:
Class: <Errno::EEXIST>
Message: <"File exists - (../sub1/real_file, output/includes/sub2/link_file)">

I already know how to solve that—by adding :remove_destination => true to the copy:

    def gather_and_process
      files = Dir.glob("book/*")
      FileUtils.cp_r files, 'output', :remove_destination => true
    end

With that, I have a bug fix pull request ready to go:

➜  git-scribe git:(test-fixes) ✗ ruby test/gen_test.rb -n test_scribe_don_t_crash_on_symlinks_when_run_twice
Loaded suite test/gen_test
Started
.

Finished in 1.607530602 seconds.

1 tests, 1 assertions, 0 failures, 0 errors, 0 pendings, 0 omissions, 0 notifications
100% passed

0.62 tests/s, 0.62 assertions/s

See? My tests run fast.

Problems with existing tests aside, this bug fix was pretty easy. Unfortunately, while fixing this, I found yet another place that my fork has diverged from upstream—this was in a completely different place in my copy. Ugh.


Day #247

No Dart Pygments for Me

‹prev | My Chain | next›

tl;dr asciidoc / a2x do not support pygmentized output in any format other than HTML.

The epub and mobi versions of Dart for Hipsters continue to lack Dart syntax highlighting. I have the correct version of Pygments installed, but git-scribe, despite throwing all kinds of warnings before the pygments update, is still not producing epubs with syntax highlighted.

Of course, this could just be my fork of git-scribe that is causing the troubles. It has been a while, but one of the additions to my fork was a post-generate clean-up with Calibre. Perhaps this is somehow scrubbing the syntax highlighting?

The answer to that question was no. Not wanting to introduce yet another dependency to git-scribe, I made the post-generate clean-up optional by virtue of the presence of a script in the source directory for my book:

    def do_mobi
      # ....
      info "GENERATING MOBI"
      # Generate with kindlegen...
 
      cmd = @wd +  '/scripts/post-mobi.sh'
      if File.exists?(cmd) && File.executable?(cmd)
        return false unless ex(cmd)
      end
      # ...
    end

I had completely forgotten about that, which means... that Dart for Hipsters does not have the most Kindle Fire friendly mobi possible. I rectify that by copying the post-mobi.sh script from Recipes with Backbone into my dart-book repository:

➜  dart-book git:(master) ✗ mkdir -p scripts                                       
➜  dart-book git:(master) ✗ cp -p ../backbone-recipes/scripts/post-mobi.sh scripts/
➜  dart-book git:(master) ✗ cat scripts/post-mobi.sh 
#!/bin/sh

echo -n "doing post mobi things..."
ebook-convert book.mobi book_ebok.mobi --chapter-mark=none --page-breaks-before='/'
mv book.mobi book.mobi.pre-calibre
mv book_ebok.mobi book.mobi
echo "done!"

With that I can regenerate the mobi and it will now play nice with the Kindle Fire:

➜  dart-book git:(master) ✗ git-scribe gen mobi
...
GENERATING MOBI
  adding: etype (stored 0%)
  adding: META-INF/ (stored 0%)
  adding: META-INF/container.xml (deflated 33%)
...
**************************************************
* Amazon.com kindlegen(Linux)   V1.2 build 33307 *
* A command line e-book compiler                 *
* Copyright Amazon.com 2011                      *
**************************************************
...
Info(prcgen): The document identifier is: "Dart_for_Hipsters"
Info(prcgen): The file format version is V6
Info(prcgen): Saving MOBI file
Info(prcgen): MOBI File successfully generated!
doing post mobi things...
1% Converting input to HTML...
...
MOBI output written to /home/cstrom/repos/dart-book/output/book_ebok.mobi
Output saved to   /home/cstrom/repos/dart-book/output/book_ebok.mobi
done!

If you have a Kindle Fire, you can re-download Dart from Hipsters (from the same URL in the original email).

But that still leaves me with my non-syntax-highlighted problem, which clearly is not being caused by Calibre. The do_epub function in git-scribe is mercifully short:

    def do_epub
      return true if @done['epub']

      info "GENERATING EPUB"

      generate_docinfo
      # TODO: look for custom stylesheets
      cmd = "#{a2x_wss('epub')} -a docinfo -k -v #{BOOK_FILE}"
      return false unless ex(cmd)

      @done['epub'] = true
    end

Hrm... that TODO note looks promising. The a2x_wss function calls the AsciiDoc executable a2x with a hard-coded stylesheet:

    def a2x_wss(type)
      a2x(type) + " --stylesheet=stylesheets/scribe.css"
    end

    def a2x(type)
      "a2x -f #{type} -d book "
    end

Looking through the HTML in resulting epub, I see no evidence of syntax highlighting. The intermediary Docbook XML does, at least, mention Dart as the syntax type:

<simpara>We start our Dart application by loading a couple of Dart libraries with a <literal>main()</literal> function in <literal>scripts/comis.dart</literal>:</simpara>
<programlisting 
  language="dart" 
  linenumbering="unnumbered">
#import('dart:html');
#import('dart:json');

main() {
  load_comics();
}

load_comics() {
  // Do stuff here
}</programlisting>

Taking a step back, I run the simplest asciidoc / a2x command possible—just asciidoc with no command line switches—against a small, self contained chapter from my book:

➜  tmp git:(master) ✗ asciidoc -v dom.asc
asciidoc: reading: /etc/asciidoc/asciidoc.conf
...
asciidoc: writing: /home/cstrom/repos/dart-book/tmp/dom.html
asciidoc: dom.asc: line 30: filtering: pygmentize -f html -l dart  -O encoding=UTF-8
asciidoc: dom.asc: line 42: filtering: pygmentize -f html -l dart  -O encoding=UTF-8
asciidoc: dom.asc: line 53: filtering: pygmentize -f html -l dart  -O encoding=UTF-8
asciidoc: dom.asc: line 65: filtering: pygmentize -f html -l javascript  -O encoding=UTF-8
...

That looks promising. And, in fact, it does produce Dart (and Javascript) syntax highlighted output:

But I'm not using the asciidoc command to produce things from AsciiDoc format. Instead, as mentioned earlier, I am using the a2x wrapper for asciidoc. So I generate HTML from a2x using the simplest command-line options possible:

➜  tmp git:(master) ✗ a2x -f xhtml -v dom.asc
a2x: args: ['-f', 'xhtml', '-v', 'dom.asc']
a2x: executing: /usr/bin/asciidoc --backend docbook  --verbose  --out-file /home/cstrom/repos/dart-book/tmp/dom.xml /home/cstrom/repos/dart-book/tmp/dom.asc
asciidoc: reading: /etc/asciidoc/asciidoc.conf
asciidoc: reading: /home/cstrom/.asciidoc/asciidoc.conf
...
asciidoc: writing: /home/cstrom/repos/dart-book/tmp/dom.xml
a2x: executing: xmllint --nonet --noout --valid /home/cstrom/repos/dart-book/tmp/dom.xml
a2x: chdir /home/cstrom/repos/dart-book/tmp
a2x: executing: xsltproc  --stringparam callout.graphics 0 --stringparam navig.graphics 0 --stringparam admon.textlabel 1 --stringparam admon.graphics 0  --output /home/cstrom/repos/dart-book/tmp/dom.html /etc/asciidoc/docbook-xsl/xhtml.xsl /home/cstrom/repos/dart-book/tmp/dom.xml
a2x: chdir /home/cstrom/repos/dart-book/tmp
a2x: finding resources in: /home/cstrom/repos/dart-book/tmp/dom.html
a2x: finding resources in: /home/cstrom/repos/dart-book/tmp/dom.html
a2x: deleting /home/cstrom/repos/dart-book/tmp/dom.xml

There is no mention of pygmentize and there is no longer syntax highlighting in the output:

My guess is that the --backend docbook option that a2x supplies to asciidoc is the culprit here (since nothing else in the printed command line looks at all different that my first run). And, indeed, running the command-line option without that switch does produce highlighted output.

At this point, I have reached an impasse. I cannot produce pygmentized output from a2xunless it is based on HTML and no other highlighter supports pygments. At the same time, I cannot produce epub unless it is based on DocBook. Indeed, the asciidoc documentation says as much:

You also have the option of using the Pygments syntax highlighter for xhtml11 outputs.

I do not believe that I ever paid much attention that last caveat. I'm paying attention now.



Day #346

The Pypi Pygments Page in the Cheese Shop

‹prev | My Chain | next›

Last night I tried unsuccessfully to install a version of pygments capable of Dart syntax highlighting. Ultimately, I want to use this to add syntax highlighting to the epub and mobi versions of Dart for Hipsters. The git-scribe toolchain that I use for ebooks can use pygments if configured correctly (as I found last night).

Dart is new enough that it is not in version 1.4 of pygments, but seems to be in 1.5. If only I can figure out how to install it. Ah... python.

I found that easy_install Pygments simply installs version 1.4, which I already have via Ubuntu packages. The only mention of 1.5 on the download page is that I can find it in the "cheese shop". As far as I can tell, the "cheese shop" is a reference to the Monty Python sketch, but how that relates to a site named http://pypi.python.org, I cannot guess. No matter, I'm sure Ruby seems just as opaque to Pythonistas.

On the pypi pygments page (yeah, I just wrote that), it seems that I need to supply the ==dev argument. I am unsure if ==dev is a common pypi thing or not. Documentation would seem to suggest not—that supplying version numbers is more common (e.g. easy_install PackageName==1.2.3). Something to try if ==dev does not work...

While I am at it, one other thing that I cannot figure out on that pypi Pygments page is what a "tip" is:

The Pygments tip is installable with easy_install Pygments==dev

Is Ruby obscure like this? Anyhow...

I try installing the dev version:

➜  ~  sudo easy_install Pygments==dev
Searching for Pygments==dev
Reading http://pypi.python.org/simple/Pygments/
Reading http://pygments.org/
Reading http://pygments.pocoo.org/
Best match: Pygments dev
Downloading http://trac.pocoo.org/repos/pygments/trunk#egg=Pygments-dev
error: Can't download http://trac.pocoo.org/repos/pygments/trunk: 404 Not Found

Rooting through that pocoo.org URL space, it seems that Trac has been abandoned for bitbucket. I am not sure how to go about troubleshooting that, so I try ==1.5 instead:

➜  ~  sudo easy_install Pygments==1.5
Searching for Pygments==1.5
Reading http://pypi.python.org/simple/Pygments/
Reading http://pygments.org/
Reading http://pygments.pocoo.org/
Best match: Pygments 1.5
Downloading http://pypi.python.org/packages/2.7/P/Pygments/Pygments-1.5-py2.7.egg#md5=8ea02c2eafb2580336b7637be629f7ce
Processing Pygments-1.5-py2.7.egg
creating /usr/local/lib/python2.7/dist-packages/Pygments-1.5-py2.7.egg
Extracting Pygments-1.5-py2.7.egg to /usr/local/lib/python2.7/dist-packages
Adding Pygments 1.5 to easy-install.pth file
Installing pygmentize script to /usr/local/bin

Installed /usr/local/lib/python2.7/dist-packages/Pygments-1.5-py2.7.egg
Processing dependencies for Pygments==1.5
Finished processing dependencies for Pygments==1.5

That worked?! Er... I mean of course that worked. Yay!

And, indeed, I do have version 1.5 now and, more importantly, Dart pygments:

➜  ~  pygmentize -V 
Pygments version 1.5, (c) 2006-2011 by Georg Brandl.
➜  ~  pygmentize -L | grep -i dart
* dart:
    Dart (filenames *.dart)

Now when I run git-scribe gen epub, I no longer get dozens of warnings about Dart not being supported. Rather I get a single warning about a missing pygments.css stylesheet:

➜  dart-book git:(master) ✗ git-scribe gen epub
GENERATING EPUB
GENERATING HTML
asciidoc: WARNING: book.asc: line 4: include file not found: /home/cstrom/repos/dart-book/output/stylesheets/pygments.css
asciidoc: reading: /etc/asciidoc/asciidoc.conf
...

Viewing the resultant epub in iBooks, I do not have syntax highlighting, so I suspect this missing CSS file. I grab the first gist I can find that might contain useful CSS and paste it into /home/cstrom/repos/dart-book/output/stylesheets/pygments.css. The next time I run git-scribe, I no longer get any pygments-related warning, but I still do not have syntax highlighting on the iPad.

That will serve as a stopping point for tonight. I have pygments working well enough (or at least not crashing). I will examine the rest of the toolchain in an effort to get syntax highlighting in epub working. Tomorrow.

Day #345

Dart Syntax Highlighting Not Working in Git-Scribe

‹prev | My Chain | next›

Don't think that I haven't noticed that Dart for Hipsters lacks syntax highlighting for my Dart code. Like last night's bug, this was one of those things for which I just could not spare 30 minutes of investigation. Well now I can.

This is different than my issues with PDF highlighting, which I will try another day. Rather this is for the syntax highlighting in the EPUB and mobi versions.

By default, git-scribe (my ebook toolchain of choice) uses source-hightlight to do this. The problem is that source-highlight has no idea about Dart. Pygments, on the other hand were made Dart aware a while back. Perhaps that has even made it into mainstream pygments.

When I run git-scribe gen epub I am getting many unknown Dart errors from source-highlight:

....
asciidoc: WARNING: testing.asc: line 133: filter non-zero exit code: source-highlight -f xhtml -s dart: returned 1
asciidoc: WARNING: testing.asc: line 133: no output from filter: source-highlight -f xhtml -s dart
source-highlight: missing feature: language inference requires input file
source-highlight: could not find a language definition for dart
asciidoc: WARNING: testing.asc: line 159: filter non-zero exit code: source-highlight -f xhtml -s dart: returned 1
asciidoc: WARNING: testing.asc: line 159: no output from filter: source-highlight -f xhtml -s dart
source-highlight: missing feature: language inference requires input file
source-highlight: could not find a language definition for dart
....

To switch to pygments, I ought to be able to edit ~/.asciidoc/asciidoc.conf to contain:

pygments=

But this has no effect. So, instead, I add that line (or rather uncomment it) in the /etc/asciidoc/asciidoc.conf system configuration file.

Only that still seems Dart-unaware. To install a more recent version of pygments, I first install python-setuptools:

sudo apt-get install python-setuptools python-dev build-essential

This should give me easy_install for the latest and greatest pygments:

➜  ~  sudo easy_install Pygments                                       
Searching for Pygments
Best match: Pygments 1.4
Adding Pygments 1.4 to easy-install.pth file
Installing pygmentize script to /usr/local/bin

Using /usr/lib/python2.7/dist-packages
Processing dependencies for Pygments
Finished processing dependencies for Pygments

Only that still does not have Dart:

➜  ~  /usr/local/bin/pygmentize -L | grep -i perl
* antlr-perl:
    ANTLR With Perl Target (filenames *.G, *.g)
* perl, pl:
    Perl (filenames *.pl, *.pm)
* perldoc:
    Style similar to the style used in the perldoc code blocks.
➜  ~  /usr/local/bin/pygmentize -L | grep -i dart

Bah! It looks as though I will need to install pygments from source code management. That is a task better suited for tomorrow.

Day #343

Git-scribe gen 2x

‹prev | My Chain | next›

When last I worked on git-scribe, I had to rather hastily abandon my efforts to rebase my branch onto the upstream master. I had hoped to get the upstream syntax highlighting working with the resizing, cover image, and various other tweaks of mine. But I was past due for getting started with Dart for Hipsters, so I had to set it aside.

And tonight, I leave it set aside.... at least for a bit.

There is an incredibly annoying bug in my fork that I have been working around for 3 months and today I finally fix it. Whenever I update one of the ebook formats, I get something like:

➜  dart-book git:(master) git-scribe gen pdf                                                   
Error: File exists - (../jade/bin/jade, output/includes/websockets/node_modules/.bin/jade)

To get this to work, I need to remove that entire node_modules directory. Just removing the problem file is not sufficient:

➜  dart-book git:(master) git-scribe gen pdf
Error: File exists - (../jade/bin/jade, output/includes/websockets/node_modules/.bin/jade)
➜  dart-book git:(master) rm output/includes/websockets/node_modules/.bin/jade
➜  dart-book git:(master) git-scribe gen pdf
Error: File exists - (../express/bin/express, output/includes/websockets/node_modules/.bin/express)
➜  dart-book git:(master) rm -rf output/includes/websockets/node_modules/.bin
➜  dart-book git:(master) git-scribe gen pdf
Error: File exists - (jade.js, output/includes/websockets/node_modules/jade/lib/index.js)
➜  dart-book git:(master) rm -rf output/includes/websockets/node_modules
➜  dart-book git:(master) git-scribe gen pdf
GENERATING PDF
GENERATING DOCBOOK
asciidoc: reading: /etc/asciidoc/asciidoc.conf
asciidoc: reading: /home/cstrom/.asciidoc/asciidoc.conf
asciidoc: reading: /etc/asciidoc/asciidoc.conf
...
Making portrait pages on USletter paper (7.5inx9in)

But I need to do that every time. Well no more, dammit.

Rooting through git-scribe, I am reasonably sure that preparing the output directory is the source of my woes. So I add verbose to the FileUtils.cp_r call:

    def prepare_output_dir(dir='output')
      Dir.mkdir(dir) rescue nil
      FileUtils.cp_r "#{@wd}/book/.", dir, :verbose => true
      # ...
    end

I still get the failure, but now I have a system command to try:

➜  dart-book git:(master) git-scribe gen pdf     
cp -r /home/cstrom/repos/dart-book/book/. output
Error: File exists - (../jade/bin/jade, output/./includes/websockets/node_modules/.bin/jade)

Executing that cp.... works just fine:

➜  dart-book git:(master) cp -r /home/cstrom/repos/dart-book/book/. output

Eventually, I realize that the problem files are symbolic links that ruby's cp_r is not handling correctly:

➜  dart-book git:(master) find book -type l 
book/includes/websockets/node_modules/.bin/jade
book/includes/websockets/node_modules/.bin/express
book/includes/websockets/node_modules/jade/lib/index.js
➜  dart-book git:(master) find book -type l -ls
 35754    0 lrwxrwxrwx   1 cstrom   cstrom         16 Mar 12 23:23 book/includes/websockets/node_modules/.bin/jade -> ../jade/bin/jade
 35929    0 lrwxrwxrwx   1 cstrom   cstrom         22 Mar 12 23:23 book/includes/websockets/node_modules/.bin/express -> ../express/bin/express
 35118    0 lrwxrwxrwx   1 cstrom   cstrom          7 Mar 12 23:23 book/includes/websockets/node_modules/jade/lib/index.js -> jade.js

If I manually remove them, then the git-scribe gen works:

➜  dart-book git:(master) find output -type l -exec rm {} \; -print
output/includes/websockets/node_modules/.bin/jade
output/includes/websockets/node_modules/.bin/express
output/includes/websockets/node_modules/jade/lib/index.js
➜  dart-book git:(master) git-scribe gen pdf                       
cp -r /home/cstrom/repos/dart-book/book/. output
GENERATING PDF
GENERATING DOCBOOK
...
Making portrait pages on USletter paper (7.5inx9in)

I include test servers in certain directories to test the code that gets included into the book. This particular problem was for the web sockets chapter in Dart for Hipsters. This seems a legit thing to do, so I definitely need to address the problem in git-scribe, not in how I maintain my book.

To resolve this I could remove the symbolic links in ruby as well. That is a hassle and I can just as easily tell cp_r to remove all files before copy:

    def prepare_output_dir(dir='output')
      Dir.mkdir(dir) rescue nil
      FileUtils.cp_r "#{@wd}/book/.", dir, :remove_destination => true

      Dir.mkdir("#{dir}/stylesheets") rescue nil
      FileUtils.cp_r File.join(SCRIBE_ROOT, 'stylesheets'), dir
    end

That does the trick. Yay!

Looking through the stdlib ruby library, I think that :remove_destination => true works on files, not on directories. Either way, it should not affect git-scribe or my workflow. Still, that eases my conscious a bit.

With that, I can generate my book as many times as I want. I am so freaking happy.


Day #342

Git-Scribe Syntax Highlighting on Ubuntu

‹prev | My Chain | next›

I had a couple of surprises while playing with the latest version of git-scribe yesterday. First, I was able to generate different size output (7.5in x 9in) (that did not, in fact, work -- I was using the wrong gem). Second, I could not get code syntax highlighting working.

I tried both over a few months back and failed to get the re-sizing working, but definitely got the syntax highlighting working:

Unfortunately, I seem to have deleted the spike that allowed that syntax highlighting to work.

The commit that added syntax highlighting to upstream hard-coded the xslt highlighting library to the most likely installation location under OSX. Since I am on Ubuntu, I need to switch the location of that file accordingly:

<!-- xsl:param name="highlight.xslthl.config">file:///usr/local/Cellar/docbook/5.0/docbook/xsl/1.76.1/highlighting/xslthl-config.xml</xsl:param -->
<xsl:param name="highlight.xslthl.config">file:///usr/share/xslthl/highlighters/xslthl-config.xml</xsl:param>

But that has no effect—there is still no syntax highlighting present in the resultant PDF.

There are at least three locations for xslthl-config.xml on my system:

➜  ~  find /usr/share | grep xslthl-config.xml
/usr/share/xslthl/highlighters/xslthl-config.xml
/usr/share/xml/docbook/stylesheet/docbook-xsl/highlighting/xslthl-config.xml
/usr/share/xml/docbook/stylesheet/docbook-xsl-ns/highlighting/xslthl-config.xml

None of them do the trick.

My next step is to switch back to the last commit in the upstream gem:

➜  git-scribe git:(upstream) ✗ git stash save
Saved working directory and index state WIP on upstream: dd5cce2 bump to 0.1.1
HEAD is now at dd5cce2 bump to 0.1.1

With that, I am greeted with:

➜  backbone-recipes git:(master) ✗ git scribe gen pdf
GENERATING PDF
GENERATING DOCBOOK

Making portrait pages on A4 paper (210mmx297mm)

[warning] /usr/bin/fop: Unable to locate servlet-api in /usr/share/java
log4j:WARN No appenders could be found for logger (org.apache.fop.util.ContentHandlerFactoryRegistry).
log4j:WARN Please initialize the log4j system properly.

That error is apparently known by some. Following this documentation, I edit /usr/bin/fop on my system so that the last line in there reads:

run_java -Dorg.apache.commons.logging.Log=org.apache.commons.logging.impl.SimpleLog $HEADLESS org.apache.fop.cli.Main "$@"

Fixing that, I see:

➜  backbone-recipes git:(master) ✗ git scribe gen pdf
GENERATING PDF
GENERATING DOCBOOK

Making portrait pages on A4 paper (210mmx297mm)
Loading Xslthl configuration from file:///usr/share/xslthl/highlighters/xslthl-config.xml...
[warning] /usr/bin/fop: Unable to locate servlet-api in /usr/share/java
[ERROR] FOP - Exception javax.xml.transform.TransformerException: org.apache.fop.fo.ValidationException: file:/home/cstrom/repos/backbone-recipes/output/book.fo:1:108583: Error(1/108583): fo:page-sequence is not a valid child element of fo:flow.
        at org.apache.fop.cli.InputHandler.transformTo(InputHandler.java:217)
        at org.apache.fop.cli.InputHandler.renderTo(InputHandler.java:125)
        at org.apache.fop.cli.Main.startFOP(Main.java:166)
        at org.apache.fop.cli.Main.main(Main.java:197)
Caused by: javax.xml.transform.TransformerException: org.apache.fop.fo.ValidationException: file:/home/cstrom/repos/backbone-recipes/output/book.fo:1:108583: Error(1/108583): fo:page-sequence is not a valid child element of fo:flow.
        at org.apache.xalan.transformer.TransformerIdentityImpl.transform(TransformerIdentityImpl.java:502)
        at org.apache.fop.cli.InputHandler.transformTo(InputHandler.java:214)
        ... 3 more
Caused by: org.apache.fop.fo.ValidationException: file:/home/cstrom/repos/backbone-recipes/output/book.fo:1:108583: Error(1/108583): fo:page-sequence is not a valid child element of fo:flow.
        at org.apache.fop.fo.FONode.invalidChildError(FONode.java:435)
        at org.apache.fop.fo.FONode.invalidChildError(FONode.java:420)
        at org.apache.fop.fo.pagination.Flow.validateChildNode(Flow.java:105)
...

Ugh. I think it is starting to come back to me. I believe that I ended up hand editing the fo output this fall. I don't think that is a viable long term solution, so I call it a night here to ruminate on next steps...

Update: I eventually track down the FO / page-sequence errors to the preface. If I eliminate the preface, I again get PDF with syntax highlighting:

But now I have two problems on my hands: the page size is again wrong and I cannot include preface chapters in my PDF anymore ugh. That is definitely something on which to ruminate.

Day #147

Tech PDFs should be 7.5" x 9"

‹prev | My Chain | next›

I have made a right old mess out of my fork of the git-scribe tool chain for producing ebooks. A mess it may be, but the resultant mobi is a bit better in that code blocks no longer split across pages and some Kindles better recognize the result as an ebook.

But really, I should do what I can to get back to upstream. Most of the work that I have in my fork is better epub/mobi out, but there is at least one PDF fix that I consider a show-stopper. Technical PDFs should be 7.5" x 9". Go on, check your PDFs from "real" publishers. That's how big they are. 8.5" x 11" is fine for printing on US-letter, but it is just too darn tall for computer reading.

Anyhow, I had got 7.5"x9" working back on my original fork fairly easily. Let's see how well that applies to recent upstream changes.

➜  git-scribe git:(master) git fetch upstream
➜  git-scribe git:(master) git co -b upstream -t upstream/master
Branch upstream set up to track remote branch master from upstream.
Switched to a new branch 'upstream'

The most obvious thing to try is cherry picking my old commit into this upstream branch:

➜  git-scribe git:(upstream) git cherry-pick b94110f
error: could not apply b94110f... Certain publishers prefer 7.5x9 inch paper size
hint: after resolving the conflicts, mark the corrected paths
hint: with 'git add ' or 'git rm '
hint: and commit the result with 'git commit'

The conflict is in the lib/generate.rb file responsible for, well, just about everything in an ebook generating library:

➜  git-scribe git:(upstream) ✗ gst
# On branch upstream
# Changes to be committed:
#
#       modified:   docbook-xsl/fo.xsl
#
# Unmerged paths:
#   (use "git add/rm <file>..." as appropriate to mark resolution)
#
#       both modified:      lib/git-scribe/generate.rb
#

More specifically, it looks as though the xsltproc command-line is missing the page-size options:

      java_options = {
        'callout.graphics' => 0,
        'navig.graphics'   => 0,
        'admon.textlabel'  => 1,
        'admon.graphics'   => 0,
      }
      run_xslt "-o #{local('book.fo')} #{local('book.xml')} #{base('docbook-xsl/fo.xsl')}", java_options

Back in my version, I had added the page-size:

strparams = {'callout.graphics' => 0,
                   'navig.graphics' => 0,
                   'admon.textlabel' => 1,
                   'admon.graphics' => 0,
                   'page.width' => '7.5in',
                   'page.height' => '9in'
      }
      param = strparams.map { |k, v| "--stringparam #{k} #{v}" }.join(' ')
      cmd = "xsltproc  --nonet #{param} --output #{local('book.fo')} #{base('docbook-xsl/fo.xsl')} #{local('book.xml')}"
      ex(cmd)

The easiest thing to try is to simply add those options to the new version:

java_options = {
        'callout.graphics' => 0,
        'navig.graphics'   => 0,
        'admon.textlabel'  => 1,
        'admon.graphics'   => 0,
        'page.width' => '7.5in',
        'page.height' => '9in'
      }
      run_xslt "-o #{local('book.fo')} #{local('book.xml')} #{base('docbook-xsl/fo.xsl')}", java_options
      ex "fop -fo #{local('book.fo')} -pdf #{local('book.pdf')}"

After re-building and re-installing the gem, I give it another try:

➜  backbone-recipes git:(master) ✗ git-scribe gen pdf
GENERATING PDF
GENERATING DOCBOOK
asciidoc: reading: /etc/asciidoc/asciidoc.conf
asciidoc: reading: /home/cstrom/.asciidoc/asciidoc.conf
asciidoc: reading: /etc/asciidoc/asciidoc.conf
asciidoc: reading: /home/cstrom/.asciidoc/asciidoc.conf
asciidoc: reading: /home/cstrom/repos/backbone-recipes/output/book.asc
asciidoc: reading: /etc/asciidoc/docbook45.conf
asciidoc: reading: /etc/asciidoc/filters/graphviz/graphviz-filter.conf
asciidoc: reading: /etc/asciidoc/filters/music/music-filter.conf
asciidoc: reading: /etc/asciidoc/filters/code/code-filter.conf
asciidoc: reading: /etc/asciidoc/filters/source/source-highlight-filter.conf
asciidoc: reading: /etc/asciidoc/filters/latex/latex-filter.conf
asciidoc: reading: /etc/asciidoc/lang-en.conf
asciidoc: writing: /home/cstrom/repos/backbone-recipes/output/book.xml
asciidoc: book.asc: line 7: reading: /home/cstrom/repos/backbone-recipes/output/history.asc
asciidoc: book.asc: line 16: reading: /home/cstrom/repos/backbone-recipes/output/introduction/namespacing.asc
Making portrait pages on USletter paper (7.5inx9in)
...

And it works! I could have sworn that I tried that a couple months back without success.

Just to be sure, I check the properties of the resultant PDF and, sure enough, it is 7.5" x 9":

I call it a night here. Up tomorrow, I need to see if I can get syntax highlighting working again with upstream. I had it working a few months ago, but am unable to find the magical combination of configuration options and installed packages to get it working tonight.

Update: Ugh. It turns out that I was re-installing my old version (0.0.9) of the gem rather than the new version (0.1.1). And so, page width and height are not, in fact, being honored.

Day #246

Post-Mobi Clean-up in Git-Scribe

‹prev | My Chain | next›

Running the mobi generated by git-scribe through a Calibre command-line conversion now seems like a useful thing. The resultant mobi is an "EBOK" ebook (rather than a "PDOC" personal document) which works better on some Kindles. I believe that I have resolved any display issues from the resultant Calibre mobi, so there is no reason not to use it.

No reason not to switch to it save that Calibre's command line options do not include the "EBOK" setting. Rather, it needs to be configure via the GUI. That is a not a recipe for a successful pull request back to upstream.

I think, for now, that I will introduce a post-mobi build step to the mobi generation. In my fork of git-scribe, I alter the do_mobi to optionally execute a shell script if present in the current working directory:

   def do_mobi
      return true if @done['mobi']

      do_epub

      info "GENERATING MOBI"

      decorate_epub_for_mobi

      cmd = "kindlegen -verbose book_for_mobi.epub -o book.mobi"
      return false unless ex(cmd)

      cmd = @wd +  '/scripts/post-mobi.sh'
      if File.exists?(cmd) && File.executable?(cmd)
        return false unless ex(cmd)
      end

      @done['mobi'] = true
    end

I rebuild the gem and install it:

➜  git-scribe git:(master) ✗ gem build git-scribe.gemspec  
...
➜  git-scribe git:(master) ✗ gem install git-scribe-0.0.9.gem
Successfully installed git-scribe-0.0.9
1 gem installed

Then, in the Recipes with Backbone source, I create scripts/post-mobi.sh to include:

#!/bin/sh

echo "doing post mobi things..."
ebook-convert book.mobi book_ebok.mobi --chapter-mark=none --page-breaks-before='/'
echo "done!"

Now, when I run git scribe gen mobi, I see entirely too much output (from git-scribe, from the dependent kindlegen tool and from Calibre's ebook-convert, but in there, I do see my local script echo statements:

➜  backbone-recipes git:(master) ✗ git-scribe gen mobi
...
doing post mobi things...
Converting input to HTML...
InputFormatPlugin: MOBI Input running
on /home/cstrom/repos/backbone-recipes/output/book.mobi
Parsing all content...
Forcing Recipes_with_Backbone.html into XHTML namespace
34% Running transforms on ebook... 
Merging user specified metadata... 
Detecting structure...
        Detected chapter: Chapter 1. Namespacing
Flattening CSS and remapping font sizes...
Source base font size is 12.00000pt
Removing fake margins...
Cleaning up manifest...
Trimming unused files from manifest...
Trimming 'images/00002.jpg' from manifest
Creating MOBI Output...
67% Creating MOBI Output
Generating in-line TOC...
Applying case-transforming CSS...  
Rasterizing SVG images...
Converting XHTML to Mobipocket markup...
Serializing markup content...
  Compressing markup content...
Generating flat CTOC ...
  CNCX utilization: 1 record, 0% full
Indexing navPoints ...
Generating INDX ...
Serializing images...
MOBI output written to /home/cstrom/repos/backbone-recipes/output/book_ebok.mobi
Output saved to   /home/cstrom/repos/backbone-recipes/output/book_ebok.mobi
done!

That will do to clean up my personal toolchain. Unfortunately, I am getting further and further away from the upstream version of git-scribe. I think tomorrow I shall have a look to see what it would take to get me back on track.

Day #245

Too Many Page Breaks in Calibre Mobi

‹prev | My Chain | next›

As of yesterday, I am strongly considering adding Calibre to my already hacked up version of the git-scribe toolchain for producing ebooks. This seems like a good idea so that I can get "Book" support on the Kindle Fire. Calibre is able to add a Mobi header (cdetype=EBOK) that the Fire recognizes as a book rather than a "personal document".

A quick glance through the resulting book on the Fire, however, reveals a show-stopper for me: there are page breaks before every heading in the book. In Recipes with Backbone, this means that our 1-2 sentence introductions now appear on a nearly blank page. The mobi generated by git-scribe does not suffer from this problem, so it must be Calibre that is adding the page breaks.

I try reducing the page breaks from h1 and h2 tags on the command line:

ebook-convert book.mobi book_ebok.mobi --chapter-mark=none --page-breaks-before="//*[name()='h1']"

but this seems to have no effect.

So I take a look at the various mobis in the Calibre display tool. For both the git-scribe and the Calibre versions of the document, the display in the viewer looks good:

Taking a look at the git-scribe DOM (by inspecting the element in Calibre's viewer), I see just the one manual page break (mpb_pagebreak):

By contrast, the Calibre DOM has manual page breaks before each sub-heading in the book:

It is pretty clear from the git-scribe DOM that the problem is all of the headings are h1 tags—even the sub-headings inside the chapters. Rather than dig through the toolchain to see where this is occurring (likely somewhere in asciidoc), I change the command line option to insert no page breaks:

ebook-convert book.mobi book_ebok.mobi --chapter-mark=none --page-breaks-before='/'

With that, my DOM only retains the original manual page break before the chapter start and no in-chapter breaks:

Hopefully that does it for the Calibre clean-up. I will root through Recipes with Backbone a little more today. If all looks good, I will add a post-mobi.sh script to my git-scribe tool chain tomorrow.


Day #144

Command Line Calibre

‹prev | My Chain | next›

I spend a good chunk of time today researching. I had hoped to find an easy solution that would allow me to create and modify the "exth" header in mobi files. If I can just do that, then I can set the "cdetype" in that header to "EBOK", which would signal to Kindles that books like Recipes with Backbone are ebooks and not "personal documents" ("PDOC").

Last night I was able to get this working with the GUI tool Calibre. The problem is that it is a GUI tool. A big appeal of git-scribe is that automates so much of the ebook chain. In my fork of git-scribe, I have it to the point that I can generate PDF, epub and mobi versions of a book and zip them all into a single file with just one command. I do not relish adding a manual, GUI step to that process.

The best I can come up with for command-line solutions is mobiperl. It seems like it would do the trick -- it was the inspiration for Calibre's EBOK support. But I am not going to pursue that. There are already an alarming number of dependencies in git-scribe, I do not think adding perl and all of mobiperl's requirements is going to help.

In ruby-land (git-scribe is written in ruby), there is the mobi gem, but that is for reading mobi, not writing. And it does not seem to be exth-aware. I could try writing my own, but that just feels like too much of a rabbit hole.

So it seems as though I am stuck with Calibre. It turns out that there is a set of command-line tools that are installed along with the GUI. One in particular, ebook-meta seems like it could be of some use, except... it does not support updating "cdetype" field in a mobi.

Another command-line tool installed along with Calibre is ebook-convert. It does not support editing the cdetype either, but I have already used the GUI to always generate EBOK output.

So I give the command a try, converting the mobi version of Recipes with Backbone to... mobi:

➜  output git:(master) ebook-convert book.mobi book_ebok.mobi
1% Converting input to HTML...
InputFormatPlugin: MOBI Input running
on /home/cstrom/repos/backbone-recipes/output/book.mobi
Parsing all content...
Forcing Recipes_with_Backbone.html into XHTML namespace
34% Running transforms on ebook...
Merging user specified metadata...
Detecting structure...
        Detected chapter: 1. Who Should Read this Book
        Detected chapter: 3. How this Book is Organized
        Detected chapter: Chapter 1. Writing Client Side Apps (Without Backb
        Detected chapter: Chapter 2. Writing Backbone Applications
        Detected chapter: Chapter 3. Namespacing
        Detected chapter: Chapter 4. View Templates with Underscore.js
        Detected chapter: Chapter 5. Instantiated View
        Detected chapter: Chapter 6. Collection View
        Detected chapter: Chapter 7. View Signature
        Detected chapter: Chapter 8. Fill-In Rendering
        Detected chapter: Chapter 9. Actions and Animations
        Detected chapter: Chapter 10. Reduced Models and Collections
        Detected chapter: Chapter 11. Non-REST Models
        Detected chapter: Chapter 12. Changes Feed
        Detected chapter: Chapter 13. Pagination and Search
        Detected chapter: Chapter 14. Constructor Route
        Detected chapter: Chapter 15. Router Redirection
        Detected chapter: Chapter 16. Evented Routers
        Detected chapter: Chapter 17. Object References in Backbone
        Detected chapter: Chapter 18. Custom Events
Flattening CSS and remapping font sizes...
Source base font size is 12.00000pt
Removing fake margins...
Cleaning up manifest...
Trimming unused files from manifest...
Trimming 'images/00007.jpg' from manifest
Creating MOBI Output...             
67% Creating MOBI Output
Generating in-line TOC...
Applying case-transforming CSS...
Rasterizing SVG images...
Converting XHTML to Mobipocket markup...
Serializing markup content...
  Compressing markup content...
Generating flat CTOC ...
  CNCX utilization: 1 record, 0% full
Indexing navPoints ...
Generating INDX ...
Serializing images...
MOBI output written to /home/cstrom/repos/backbone-recipes/output/book_ebok.mobi
Output saved to   /home/cstrom/repos/backbone-recipes/output/book_ebok.mobi

Hrm... it really seems to do quite a bit of changing and re-arranging of things. I had expected, based on a cursory reading of the mobi-to-mobi options for ebook-convert that it would do less.

Still, it does the job. When I copy the generated book back to my Kindle Fire, it shows up under the "Books" section rather than the "Documents" section.

Hrm... perhaps I can add an optional "clean-up" step to git-scribe that can invoke an arbitrary command-line script. I cannot explicitly include Calibre since it requires the GUI configuration from last night. But first, I think, I need to compare the output from ebook-convert with the original git-scribe mobi. I worked hard three months ago to get it just right. I will pick back up with that tomorrow.


Day #243

Kindle Fire Book vs. Document

‹prev | My Chain | next›

One of the hardest things to get right with ebook generation is the on-device experience. In part, this is because it is so time-consuming to try out even small changes. I have no idea how to speed up the process of generating the book, mounting an associated device, copying the stinking thing over to the device and then unmounting the device so that the device is back in read mode.

To be sure, sending the book to the device wirelessly is generally a better bet, but then you are at the mercy of network issues. You are never quite sure if the copy on the device is the one you just sent or the one that seemed to get lost 30 minutes ago.

I bring all of this up, because we received a report from Derick Bailey that copying a the mobi version of Recipes with Backbone onto the Kindle Fire fails to register the book as an actual book on the device. Rather it shows up as a "document".

So I regenerate the mobi version of the book:

➜  backbone-recipes git:(master) git-scribe gen mobi
GENERATING EPUB
GENERATING HTML
...
Info(prcgen): The file format version is V6
Info(prcgen): Saving MOBI file
Info(prcgen): MOBI File generated with WARNINGS!

Then I copy the output over to my Fire:

➜  backbone-recipes git:(master) cp output/book.mobi /media/KINDLE/Books 
➜  backbone-recipes git:(master) umount /media/KINDLE 

And sure enough, the book does not show up under "Books" in the Fire, but under "Documents".

But, you know what? So do all of the PragProg books that have been getting sent to the Fire via my @free.kindle.com account. So, at least I am in good company.

Even so, it would be nice if I could get that working.

Derick was kind enough to provide a pointer to get started. In that thread, it is suggested that Calibre might be able to solve the problem if I convert my output to "EBOK" (instead of "PDOC"). So, under preferences / output, I change the output from "[PDOC]" to "[EBOK]":

Then, I load the mobi version of Recipes with Backbone into Calibre, and convert it to... mobi. Again, I copy the book over to the Fire:

➜  backbone-recipes git:(master) cp ~/Calibre\ Library/Nick\ Gauthier/Recipes\ with\ Backbone\ \(30\)/Recipes\ with\ Backbone\ -\ Nick\ Gauthier.mobi /media/KINDLE/Books/book.mobi
➜  backbone-recipes git:(master) umount /media/KINDLE   

And, what do you know? It worked. Recipes with Backbone now shows up under Books on the Fire.

But needing to run the book through a manual process like Calibre is less than ideal.

Unfortunately, the kindlegen tool provided by Amazon does not support setting this value. This looks to be a binary field somewhere in the mobi, so I am hesitant even get into trying to figure this out manually. And to make matters worse, I cannot reverse engineer the mobi because the generated mobi is not unzip-able:

➜  rwb_mobi  unzip book.mobi  
Archive:  book.mobi
  End-of-central-directory signature not found.  Either this file is not
  a zipfile, or it constitutes one disk of a multi-part archive.  In the
  latter case the central directory and zipfile comment will be found on
  the last disk(s) of this archive.
unzip:  cannot find zipfile directory in one of book.mobi or
        book.mobi.zip, and cannot find book.mobi.ZIP, period.

If I do nothing, I am no worse off than other ebook vendors. Still, it would be cool to solve this. I will have to ruminate some...

Day #242

Git-Scribe Syntax Highlighting

‹prev | My Chain | next›

Yesterday I was able to cherry pick a commit from the upstream git-scribe into my fork of git-scribe that prevented code blocks from splitting across pages in the generated PDF. An even more exciting development in upstream is syntax highlight in code blocks—something I was never able to get working on my own.

Ultimately, I should try to get back on upstream, but I had invested quite a bit of time in getting the epub and mobi formatting working nicely in my fork (I did submit pull requests) before upstream went a different direction. Also, when I had tried out upstream after those changes went it, I was no longer able to control the PDF page size (I am use 7"x9"). It is going to be hard to get that rebased onto upstream if I cannot get my local a little closer.

Anyhow, I am pretty sure that I could not get highlighting working when I cherry picked the commit previously, but let's try it out to be sure....

➜  git-scribe git:(master) git cherry-pick cddb437d27e01c4fda7ae5a51bb47cdd57595acb
error: could not apply cddb437... Add syntax highlighting for PDFs
hint: after resolving the conflicts, mark the corrected paths
hint: with 'git add <paths>' or 'git rm <paths>'
hint: and commit the result with 'git commit'

Checking the conflicts, I see:

➜  git-scribe git:(master) ✗ gst
# On branch master
# Changes to be committed:
#
#       modified:   docbook-xsl/fo/highlight.xsl
#       modified:   test/gen_test.rb
#       new file:   vendor/saxon.jar
#       new file:   vendor/xslthl-2.0.2.jar
#
# Unmerged paths:
#   (use "git add/rm ..." as appropriate to mark resolution)
#
#       both modified:      .gitignore
#       both modified:      docbook-xsl/fo.xsl
#       both modified:      lib/git-scribe/generate.rb

Ah, wait a second. Now it's all coming back to me. Upstream is now using saxon (java) to convert so that it has access to the libxslthl library which is written in java. Ugh.

Before I go down that path, I would like to see if alternatives might work. To the best of my knowledge, this means pygmentize. Rather than fiddle with it directly, I give the python script in this article a try (copying the html.xslt code listing from that same article):

➜  output git:(master) docbook_build.py --html html.xslt book.xml
Note: namesp. add : added namespace before processing              Recipes with Backbone
Writing /home/cstrom/repos/backbone-recipes/output/pr01.html for preface(_history)
Writing /home/cstrom/repos/backbone-recipes/output/pr02s02.html for section(_contact_us)
Writing /home/cstrom/repos/backbone-recipes/output/pr02s03.html for section(_how_this_book_is_organized)
Writing /home/cstrom/repos/backbone-recipes/output/pr02.html for preface(_introduction)
Writing /home/cstrom/repos/backbone-recipes/output/ch01.html for chapter(chapter_non_backbone)
....
Writing /home/cstrom/repos/backbone-recipes/output/index.html for book

When I take a look at the output, I find:

Aw, dang, that's really no improvement over:

To be sure the pygmentize formatting is a little prettier, but the same things are getting highlighted: this, keywords (return, function), and strings. None of the class names are highlighted. None of the method names in either is highlighted. So there really is no benefit to sticking with the old xsltproc-based way.

So it looks as though I am going to have to follow upstream down the saxon path. I will get started with that tomorrow, but, instead of working with the syntax highlighting, I am going to see if I can solve the page size first. 8.5in x 11in just won't cut it for me.

Day #241

Code Blocks in Git-Scribe

‹prev | My Chain | next›

With my last remaining PhantomJS questions answered last night, I think I am ready to bid a fond adieu to Backbone.js. I will still have need of the occasional follow-up post, but the regular material needed to support Recipes with Backbone is complete. And so, I turn my attention back to my toolchain for generating ebooks, git-scribe.

Between the time that I wrote The SPDY Book and Recipes with Backbone, there was quite a bit of activity on git-scribe. Much of it was fairly exciting. One of the features added was one with I had wrestled unsuccessfully: code-highlighting.

The thing about highlighting is that not all highlighters are created equal. The code in our book ended up looking like:

In Emacs, by contrast, methods properties and top-level class names are highlighted:

That coupled with an unsatisfactory control over the page size led my co-author, Nick Gauthier, and I to stick with my my fork of git-scribe. I hope to take a few days now ideally to switch back over to the mainstream git-scribe. Failing that, I hope to at least get a little closer.

Tonight, I start with the getting closer. One of the annoyances in my git-scribe fork is that code sample can span pages:

That just sucks. Fortunately there's a commit for that. A commit, that is, in the upstream version of git-scribe. I am already tracking upstream, so I can just cherry-pick that into my local copy:

➜  git-scribe git:(master)  git cherry-pick 8e35785

Now, to test that out. It is not exactly the kind of thing that I can BDD since the desired output is entirely visual. So I start with the bit of code that was not wrapping previously. I delete enough lines so that it is just at the bottom of one page:

Then I add just enough text to add one line of text above that code sample and re-check the output:

Yay! The one line that would have otherwise pushed a single line of a closing parenthesis onto the next page now sends the entire code block onto the next page.

That is a small victory, but I'll take it. If nothing else, it is a good way to get me back in the swing of git-scribe coding and I can definitely make use of that immediately to improve Recipes with Backbone.


Day #240