Saturday, March 10, 2012

Converting from Markdown

I have a ZIP file containing a framework for Ema. I'll be using to redesign and refactor my current novel. It's a blank, with some templates/examples, but otherwise empty and ready to be filled in. This takes the idea of a Wiki-fied beat sheet referencing scenes and converts it in to a single Markdown file that can be processed with pandoc in to something you can tweak later -- or directly in to ebook.

Now, it is possible to convert from your beat sheet -- with each scene marked with a clear identifier -- to a single Markdown document listing all of your scenes in order. This is why the wiki beat sheet uses WikiWords for all links except links to scenes. I also recommended you include chapter titles or scene breaks at the beginning of each of the scene files.

(The pandoc documentation generates an eBook using a chapter per file -- this is required to get pandoc's table of contents working -- but because scenes may be reordered in the beat sheet before the chapters are finalized, we can't just start out with this technique. Now, "pandoc" will still work to convert to other formats, but if you want to convert directly to an ebook you'll need to split the file in to chapters after we stitch the scenes together.)

The magic happens by way of "regular expressions". These allow you to perform replacement operations leveraging parts of your earlier text.

Using Notepad++ and CMD.exe

(I used Notepad++ in my example here primarily because when it installs, it has an easy context-menu item to launch it. (And it is free and open-source.) WriteMonkey is a Windows-native Markdown editor that has regular expression support, so you should be able to use it instead of Notepad++ relatively easily. Unfortunately, MarkdownPad -- for all its flash -- totally lacks any replace support, let alone regular expression support.)

The editors bundled with Windows do not support regular expressions. Notepad++ is a free multi-file editor with regular expression support. It's also open-source (GPL license). Similar techniques should work with any number of editors.

First save a copy of your "BeatSheet.txt" to "MakeWhole.bat". Open "MakeWhole.bat" in Notepad++. You will be making changes to the file, and you do not want to lose your beat sheet.

Next you need to perform a Search -> "Replace...", then set the "search mode" to "regular expression".
In the "Find What" field: ^.*([{](.*)[}]).*$
In the "Replace To" field: type "\2.txt" >> WholeDocument.txt
Unfortunately, I didn't see a way to remove the non-title lines, so they will need to be removed by hand. Delete every line that doesn't begin with "type".

These "TYPE" commands will result in appending the files to the "WholeDocument.txt" file.

If you think you may (accidentally or not) run the script more than once, you should change the ">>" on the first line to ">". This will cause the script to overwrite the file when it starts, then append to it. (Without this change, or a "del WholeDocument.txt" added to the beginning of the file, it will keep appending the files resulting in a WholeDocument.txt file which is less than usable.)

We're not quite done, though. Ema creates files with spaces turned in to underscores, so right now the files will all be not found.

This is also easy to fix using regular expressions.
In the "Find What" field: "(.*) (.*)"
In the the "Replace To" field: "\1_\2" 
Note that each time you perform this replace, it only removes a single space from the filenames, so you may need to run it more than once.

This should leave you with a "MakeWhole.bat" file which will work to create a WholeDocument.txt file.

One idiosyncrasy of this method is that if you do not end your files with a blank line, the first octothorpe/hash (#) in the file will but up against the last paragraph of the previous file. CMD.exe doesn't have an easy method to add an empty line to the file, but you can create a file with a single blank line, then "type empty.txt" >> WholeDocument.txt between each of the scene lines if that is an issue for you.

There isn't an easy way to  split the chapters out of the WholeDocument.txt using just an editor and CMD.exe. Hey, it is Windows, you have to expect some pain. -- Or you can install CoreUtils for Windows and use csplit, as mentioned in the "Using GnuWin packages" section.

Using GnuWin packages

The GnuWin packages are free/libre software packages to provide some tools which are standard on every operating system except Windows.

We will need to install CoreUtils for Windows and sed for Windows.We do not install a shell, so these will run from CMD.exe and create a BAT file to assist.
sed -n -e 'y/ /_/' -e 's/^.*\({\(.*\)}\).*$/cat "\2.txt"\nc:\\path\\to\\echo.exe/p' < BeatSheet.txt > MakeWhole.bat
MakeWhole.bat > WholeDocument.txt
This automatically adds a newline between files. This gives you one file called "WholeDocument.txt" and it splits the chapters out in files called ChapterXX (no extension -- with XX replaced with numbers starting with 00.)

Now, I've not tested it, however I know that you will need the absolute path to the CoreUtils "echo.exe" command before it will work. Without an absolute path it will use the CMD.exe internal "echo" command (even if you specify echo.exe -- it will actually echo "exe"), Unfortunately, I do not know the path the GnuWin packages install to, so I can not provide the path. I have substituted \\path\\to\\echo.exe instead. You need to replace it with the path to "echo.exe" included in the CoreUtils for Windows package with the backslashes doubled.

If you want to split the WholeDocument.txt file in to chapters, it is as simple as:
csplit -z -b %02d.txt -f Chapter WholeDocument.txt '/^# [A-Za-z0-9]/' '{*}'
This will split the file in to chapters and keep the files with a ".txt" extension so Windows can open them.

Using GNU tools (Linux, Mac OS X, or Cygwin)
If using standard GNU-style tools from Linux, Mac OS X or Cygwin you would use (these should two commands on two lines -- they wrap in this display):
sed -n -e 'y/ /_/' -e 's/^.*\({\(.*\)}\).*$/\2.txt/p' < BeatSheet.txt | (while read FILE ; do cat "$FILE" ; echo ; done) > WholeDocument.txt
csplit -z -f Chapter WholeDocument.txt '/^# [A-Za-z0-9]/' '{*}'
This automatically adds a newline between files. This gives you one file called "WholeDocument.txt" and it splits the chapters out in files called ChapterXX (no extension -- with XX replaced with numbers starting with 00.)

Using the pandoc technique to create an ebook should now give you the table of contents, as expected.

If using ikiwiki instead of Ema

I am a big fan of ikiwiki, but the syntax is different from Ema. Since we normally use WikiWords to link documents most of the time the differences will be invisible.

(It is expected that if you're using ikiwiki, you're using Linux, Mac OS X, Cygwin or another environment which is similar. -- ikiwiki doesn't want to run in Windows.)

First, you'll want to convert the draft/empty Ema-formatted files to ikiwiki standards. To start off, we need to rename them. I like to use mmv, which would work like so:
mmv *.txt '#1.mdwn'
The difference between Ema is ikiwiki uses double-square-brackets for links to other wiki-pages [[like so]] and allows spaces in filenames. Ema uses {curly braces}, and converts spaces to underscores. Since we normally expect WikiWords to work, the only place this shows up as an issue is the beat sheet.

It is left to your discretion as to whether you want to rename my "Scene_Title.txt" (now .mdwn) file using 'mv' or 'mmv'. It is just one file, and there's just one underscore/space, so it is trivial one way or the other.

Converting the Ema-style scene titles to ikiwiki is done through a simple "sed" command:
 sed -i -e 's/{\(.*\)}/\[\[\1\]\]/g' *.mdwn
That performs an ïn-place change between the Ema-style wiki-link and the ikiwiki-style wiki-link.

Now for the differences in the compilation stage:
sed -n -e 's/^.*\(\[\[\(.*\)\]\]\).*$/\2.mdwn/p' < BeatSheet.mdwn | (while read FILE ; do cat "$FILE" ; echo ; done) > WholeDocument.mdwn
csplit -z -f Chapter WholeDocument.mdwn '/^# [A-Za-z0-9]/' '{*}'
The two differences are again, the spaces are left unmolested, and the extension is "mdwn" instead of "txt".

Saturday, February 25, 2012

Writing tools for phone (Android) and desktop (Windows)

I've previously mentioned the benefits of using future-safe notes.

On the desktop-side, I've been using Scrivener. It's a great application and I generally love it. (It stores things using XML and RTF files -- future-proof even if I need to hack together something to get the data from the XML bits.) The problem? I can only use it when I'm at my desktop. With the kids, this means I'm limited in what I can do while I'm away from my desktop.

While I've had good success writing by way of "draft" emails in Gmail on my phone and copying and pasting in to Scrivener, this only works for pounding out my word-count and totally fails when it comes to actually planning what will be written. It worked great for NaNoWriMo, but when it comes to serious planning restructuring it just doesn't work.

I tried EverNote. I had two problems with it: saving a link on my phone was slow (the app is generally slow to load) and it has a lot of features I didn't need. It's a semi-pay application, and the way they charge makes relying on it problematic without paying for it. I had existing tools which covered all other needs except link synchronizing between my phone and my desktop. In the end, I uninstalled it from my phone as it wasn't useful enough for the storage space it occupied.

I recently installed DropBox, and so I've been interested in applications that can make use of it. Link-synchronization using DropBox is easy and fast on my phone. This is particularly the case because I only care about browsing pages on my phone and saving them for future reference on my desktop. It's one-way synchronization and not two-way.

Actually, I installed DropBox because I had installed Ema Personal Wiki. It's Windows and Android software that uses Markdown for light-weight text formatting. It defaults to using DropBox to synchronize between the desktop and the phone. It's a little like ikiwiki, but it sacrifices features for a GUI. Unfortunately, Ema doesn't support directories and uses a different syntax for free-links.

Now, I have had Papyrus installed for some time. It's a super simple plain-text editor. The thing is -- using Gmail directly was generally more convenient. It's strictly offline, so copying things to/from Papyrus wasn't convenient. The addition of DropBox on my phone, means that copying things to my desktop got easier. However, if I'm using Markdown elsewhere, it makes sense to upgrade to a MarkDown-based editor.

On my phone, this led to the installation of Writer. It's a Markdown-based editor designed for writing documents -- complete with word-count functionality, and representing the text-formatting as-you-type. (Not quite WYSIWYG, you get things like *italic* and **bold**. Very useful if you find yourself sometimes using Markdown-special characters.) Writer currently has no preferences. It does not synchronize with anything, but there are plans to eventually support DropBox (and other) synchronization.

Compared to Epistle, Writer looks to be a little earlier on in the development process. Epistle -- while it doesn't have support for word-count -- supports DropBox to the point you can specify the folder to use, so you can have it access the same documents as Ema. (It doesn't do anything with the wikilinks.) You can enable "HTML preview of Markdown documents" which will get you a reasonably accurate representation of the rendered Markdown after it is saved -- but editing the document is purely plain-text, unlike Writer.

Of the two, I'm favoring Writer. If it had DropBox support with the capability to specify the DropBox folder, there would be no contest. I'm a little surprised I've not seen a clearly open-source Markdown file editor. (Ema is a personal wiki, not just a file editor. It just presents bare Markdown in a plain-text window. It would be nicer if it did some styling.)

Just a note, though, Papyrus, Writer, Epistle and Ema all load faster than EverNote did. They're all more responsive and more useful for general note-taking. Epistle (alone) supports "sharing" links. It does so with editable descriptions -- like EverNote provided -- except it is a ton faster. This combined with the fact I've the folder set to use my PersonalWiki folder will mean I'll keep Epistle installed for link-preservation if for no other reason.

With regards to ikiwiki... It's generally awesome. It's a wiki-compiler that can use Markdown as well many other light-weight formatting languages. It's plug-in based and already has more features than many competing products. It uses a real first-class version control system with all the functionality that goes along with that. It's also written in Perl and has some expectations of a Unix-like operating system.

If you used Linux or a Mac, you would get significantly more functionality if you used ikiwiki instead of Ema. Of course, to make it work with a phone would require a bit of work to get it to work well with DropBox and your phone-native Markdown editor. This would be totally doable, mind you. You need only a little more than what Ema already supports to be basically ikiwiki-compatible, and Ema is open-source. Extending it to support [[free-links]] instead of {free-links}, the .mdwn extension instead of the .txt extension, and basic directory support used by ikiwiki should be straight-forward. Most of the other ikiwiki extensions would then be handled by ikiwiki at compile-time.

Anyway, while exploring Ema, and adapting my Magic Door "beat sheet" to a format that Ema could use reasonably (it is where I need to spend some time), it occurred to me that the whole framework I use in Scrivener could be adapted toward a Markdown-based wiki. It would be better in ikiwiki, of course, but it would work in Ema. I'm going to check it out.

To convert the Scrivener notes in to something I can feed to Ema (and to convert back) I'll be using pandoc.

To create the Markdown, I've told Scrivener to export as HTML, then started off telling pandoc to convert from HTML to Markdown. (pandoc can not convert from RTF to Markdown.) The advantage here is that Markdown supports embedded HTML for things that can not be easily encoded in a lighter-weight markup. When the pandoc-generated Markdown looks incorrect, I can copy/paste directly from Scrivener's HTML.

To convert from Markdown to RTF, I'll use pandoc as well as it supports RTF as an output format. The RTF can be easily imported in to Scrivener. Of course, I'll need to verify the conversion works as expected, but the ad-hoc HTML use in Markdown is a key feature, so if that's missing it's really vastly incomplete. (And I'll have to import HTML documents in to Scrivener.)

I may not be using Ema to edit the individual files on the desktop. There are a couple other options for the (Windows) desktop. MarkdownPad is super pretty, but it looks like WriteMonkey totally wins out on features. While they both have basic word-count support, WriteMonkey supports "sprint writing" with count-down timers, a variety of customizable lookups, etc.

This is what I'm thinking so far, at least. I'll be adapting my "empty" Scrivener project with my various sections for novel development in to a set of Ema-linked Markdown-based Wiki-pages. The Ema-style-links can then be machine-converted to be usable for ikiwiki.

Monday, January 23, 2012

Story design: Snowflake vs. Architecture

I've been reading Story Engineering by Larry Brooks. The book is extremely practical and I highly recommend it for any and all would-be authors. It talks about his six core competencies of successful storytelling.

Recently, I was rereading The Snowflake Method of Story Design by Randy Ingermanson and thinking about how the two fit together. A key point that Mr. Brooks makes is that the information he provides is not new, and it is the same sort of thing all other writing advice talks about -- but most other writing advice doesn't mention the whole picture.

So the Snowflake method has nine (or ten -- but Mr. Ingermanson doesn't do the tenth step anymore) steps. Here are how they relate to the terms Mr. Brooks uses:

1. Write a killer concept. Concept is one of Mr. Brooks "core competencies of story-telling." He has a lot of articles about concept, as it is very important to get right.

The recommendations for the wording between the two descriptions are significantly different. Mr. Brooks recommends phrasing it as a "what if?" question. Mr. Ingermanson recommends a one sentence summary no longer than fifteen words. The underlying meaning is the same: This is the most abstract representation of the story, and in exploring this we get the entire story.

2. Expand the concept in to the five key moments: opening, first plot point, midpoint, second plot point, ending. These are key transition points in Larry Brooks' story structure. His story structure is based upon a four act design. (Which is the same as a three act design with the second act split down the middle.) The opening is act 1, the first plot point is the transition between act 1 and act 2. The midpoint is the transition between act 2 and act 3. The second plot point is the transition between act 3 and act 4. The ending is act 4.

At this point each of these five key moments should be just a sentence. Mr. Ingermanson uses paragraph form, but I prefer a list.

It is important to note that Mr. Ingermanson uses a traditional three act structure with the setup, three disasters and an ending for this step. These "disasters" map directly to (and are placed so as to exactly align with) Mr. Brooks' first plot point, midpoint, and second plot point. He uses the same structure, but doesn't have an explanation as to why that is the best structure.

3a. Mr. Ingermanson is a big proponent of the idea that character is the most important part of a story. (He's wrong, conflict is the most important part of a story.)

The heavy focus on characterization, means that Mr. Ingermanson is taking care of theme by way of character arc. There is likely to be a clear theme present simply due to the rich characterization, even when it wasn't explicitly defined.

You  need to know your theme. Some concepts are instantly thematic on their own, others less so. In some cases a theme is implied by character background, setting, etc.

3b. For your main characters (at least the protagonist and antagonist) you need to start working on characterization. The character's development through her or his character arc changes as the act of the overall story moves along. You want to write three-dimensional characters but don't go overboard. It is okay to have one-dimensional minor characters.

To map with Mr. Ingermanson's snowflake design, you need enough characterization that you know the conflict each main character will be facing (inner and external) and can develop a short description of the character arc for the main characters. The characterization details can -- and should -- be brief at this point as we'll come back to fill them in later.

4. Take the five key moments written in step 2, and rough out one paragraph descriptions of each of the four acts. (The second through the fifth key moment goes -- perhaps in an enhanced form -- at the end of each of the four paragraphs. The first key moment can disappear in to the first paragraph.)

5. Go back to your characterization information and add more details. Fill more in. Add a richer back-story and a more detailed character arc. At this point a brief description of the character's arc can be expanded to four sentences describing what will happen in each of the four acts of the book.

Mr. Ingermanson recommends a page for main characters and a half-page for other important characters.

6. Now take each of the four paragraphs you wrote in step 4 and expand each out to a full page. With knowledge of character arc, you can include subplots and points of characterization.

7. Go back to your characterization information and fill out any/all additional information.

8. Take the four page synopsis created in step 6, and turn it in to a beat sheet.

Mr. Ingermanson strongly recommends a spreadsheet. Mr. Brooks does not particularly recommend a spreadsheet. (His provided blank sample is a simple list.)


Mr. Ingermanson uses different language when he talks about the concept in step 1. The specific recommendations about the language are different, but the meaning of the words are the same. In both cases it is a concise description which, when explored leads to the story. The other big difference is Mr. Ingermanson doesn't mention theme at all. Reading Mr. Brooks' advice on theme, it is clear that Mr. Ingermanson is implicitly covering theme. He may not be explicitly defining it, but if you looked for it in Mr. Ingermanson's works you would probably be able to find it.

The overlap is extensive. There is only one omission (theme) and it's the one that Mr. Brooks describes as something you don't need to actively concentrate on: if you use thematic elements it takes care of itself. You only need to be aware of it in as much as you can only steer the theme if you have a place to steer it toward. The rich characterization advocated by Mr. Ingermanson would pull in thematic elements for characters even when they were not inherent in the plot or setting.

The four that mostly overlap are the "elemental" core competencies: concept, theme, character, and structure. The last two that Mr. Brooks has are "executional" and are scene construction and writing voice. The executional competencies mean that none of the elemental competencies by-pass the need for well written prose. Mr. Brooks is very clear in that you need to master all six to get your novel published.

Wednesday, January 4, 2012

DRM doesn't work and annoys readers

Is your book only available in the US Amazon store? If so, it uses DRM to prevent customers in other parts of the world from reading your book. The DRM won't work and may drive potential readers to piracy.

DRM (digital restrictions management) restricts readers ability to read your book in all regions of the world, and prevents legal fair-use. (Such fair-use may even be for reviews and fan sites -- things you may want to encourage.) With region-based restrictions it can mean it is faster and easier to import a used physical copy of a book from overseas than it is to get a legal electronic copy -- even though international shipping can be quite expensive. If it is faster and cheaper for someone to buy it illegally the chances are they will do so.  With copy-and-paste it is something that is quite legal (under fair-use) with physical books, but it is considered theft when it is done with an eBook.

The problem is that for people who actually want to steal an eBook DRM is an inconvenience at worst. Do you want to know how easy it is?

Google the following things:
  1. "keyboard mouse recorder"
  2. "screen shot keyboard shortcut"
  3. "batch processing images"
  4.  "OCR software"
None of this software will actually removed the DRM. They are all general-purpose tools that are commonly available for legal uses.

This is on-par with real DVD pirates using analog techniques to copy DVDs instead of violating the DRM. (They make exact copies of the pits on the disc -- much like they used to pirate LPs.) The DRM is a non-issue to folks actually committed to piracy. It only impacts law-abiding citizens.

To be explicit: I do not support or approve of violating current laws. The list of Google terms was provided as an example of how the reliance on DRM software is flawed. I will not help you with any questions about violating copyright law, and I will instead forward any such emails to the FBI.

Oh, and if you think that it is possible for the operating system to do something about this, I have two more terms for you to Google: "digital camera" and "tripod".

The only way to prevent illegal distribution is to not distribute it at all. Even then it isn't reliable. That said, when things are priced right and have good service associated with legitimate purchase it isn't viable to pirate.

In particular, your English-language book should be available world-wide to all readers of English. Make it easy and affordable to buy across the globe and few people will bother with piracy.

Monday, January 2, 2012

A general purpose writing tracker

I've written a general-purpose writing tracker. (You know, because I'm masterful at procrastinating.)

It is designed for a single project, but it will work with any number of days and any number of words. You can either specify an end-date or you can specify the number of days. You can also specify a "Daily Bonus Goal" to clarify that you're really hoping to write more than the minimum required every day. (You know, to support bad word-count days and days when family interferes. By default I set this to 333.33 as my daily personal goal is at least 2K words.)

It supports entering word-count both in "words written in session" (incremental) and "words written total" (total) forms. This is specified in the "Word Count Mode" setting.

It allows you to keep track of the time you write, so you can see how fast you write. You enter the start and end times and it figures out how long you were writing. (These support starting at 10pm and ending at 2am -- and it correctly marks the duration of 4 hours.) It also has a "more" section that allows you to enter the duration directly in hours -- the idea is that it would primarily be used when you have more than two writing sessions a day and you'd use it by migrating the duration/word count information to the "more" section (so even then you never need to calculate how long you were writing).

It has a functional chart, showing where you are at, where you should be, if you met your daily bonus goal every day where you would be, and the average for your overall rate. During the month of NaNo (or Camp NaNo) the number of data-points on the graph will increase as more days are added to the chart. If a day hasn't happened yet, it doesn't show on the chart.

Unfortunately, the chart doesn't work on my Android phone, but the document itself does. This is the benefit of using a Google document.

The number of rows expands based upon the day. If the project hasn't started only the first row will be shown. To get a feel for it, tweak the start date to something two weeks ago -- you'll see a lot of things auto-populate when that happens. The last entry in the list is typically the one you want to edit.

It has a few computed fields that don't show up until a project has been started.

I do just use the default number of rows (100) so if you're going to need more than 90 days, you'll need to tweak some things. This should more than fit most people's needs. If you're tracking a whole year, the changes should be straight-forward (I hope).

It was inspired by the "NaNoWriMo Word Count Tracker" from (An XSL file which also has the capacity to track morale. -- I don't track my morale because when I'm writing I am always in a good mood.)

I've thought about writing a version that could be added to using a Google Form. I started on it, but I've lost interest -- probably because I'll never actually use it.