On Writing a Book
17 Jun 2016Last December I was holding the final printed version of my German book on Elasticsearch in my hands. It's a good feeling and though there were stressful times I don't regret writing it. In this very long post I'd like to talk about what it's like to write and publish a book the traditional way.
Signing the contract
I first got approached by dpunkt.verlag in June 2013. Niko Köbler talked to an editor of the publisher at a conference, they somehow got to Elasticsearch and he recommended me as a potential author. The first call I had with the editor was very informal – we talked about search technology and the role of Elasticsearch. They were thinking about doing a book about it in the future but there were no concrete plans. We agreed to wait for half a year and then see again.
I didn't really think that something would come out of it at all but in the end of March 2014 the editor contacted me again, telling me that they wanted to do a book on Elasticsearch and if I would be interested.
I always thought that writing a book is a good thing. Not that I really worked on going that route but I somehow admired people who had done it. So I didn't have to think a lot – I wanted to do it and agreed.
Even before signing a contract I had to create an outline of the book, i.e. create a table of contents with the headlines for the different chapters and sections in the book. This is more difficult than it might sound, even when you think you know what you want to write about. You need to make sure that all the necessary topics are covered and that it's a coherent story.
This outline is then sent to several reviewers who provide feedback for the author (and very likely also assure the publisher that the author is the right person for the job). After incorporating some feedback in the outline I had to put milestone dates on the chapters. That also means defining the order the chapters will be written in, which normally is not the order they are read in (example: the first chapter in the book is the last chapter I wrote). I already expected the writing process to be a huge undertaking, taking longer than I would think. I planned on finishing writing the book in around eight months, having it released in about one year. I signed the contract in July 2014 at Java Forum Stuttgart.
Tools
I had the choice of technology for writing the book – Word, Open Office documents or LaTex. I chose LaTex that I already knew from writing my thesis and I suspected would be less problematic when it comes to formatting issues. Later I sometimes regretted the choice as I had to struggle a lot setting the system up. Only later we found out that I didn't receive all the necessary files, so these problems could have been prevented.
Nearly all of my writing was done directly in vim. I had a notebook but didn't do a lot of writing on paper (but I am writing the first version of the post you are just reading on paper). I put all the assets under version control in Git, also pushing them to a remote server for backup and sometimes copying all of it on USB sticks. You really don't want to start again from the beginning after that much work :).
I struggled a bit with the choice of tool for images. I started with some placeholder images I created using yEd, experimented with some other tools, and finally went back to yEd again. There might be more beautiful diagrams than the ones in the book but I am quite happy with the result.
The Writing Process
Writing happens on a chapter by chapter basis according to the milestone plan I created with the outline. If you're lucky like me you can find people who are willing to review the chapters for you. Otherwise the publisher will have people for reviewing the book. I contacted some people for help and saved the publishers reviewers for the final version of the book. I am especially thankful for the help of Tobias Kraft of Exensio who reviewed each and every chapter in detail. Thanks for your work Tobias!
I started writing with the second chapter that I planned to be one coherent example on how to use Elasticsearch, how analyzing and search works and so on. I was already well prepared for writing as during the time I started to work on the book I was publishing blog posts regularly, nearly on a weekly basis. I planned on establishing a writing habit, due to my work as a Freelancer I was able to reduce client projects to four days a week and have one day I can dedicate to writing alone. But even though I planned to do so it was difficult to execute. Half a year is a very long time span and you tend to have more urgent things to do – "I will catch up on writing later on" – except you don't.
When I finished the second chapter it had taken a lot longer than expected and it grew very long. Even I knew that this was not the quality to be published in a book. And the feedback of my editor and the reviewers was the same. Though they could see where I wanted to go my writing wasn't good yet. The sentences were too long, you could notice the different times I had written the parts and all in all the chapter was far too long for an initial example. Back to the writing desk.
I was already late and had to redo everything. But after a while it got better: I extracted two more chapters on topics I didn't originally plan to write about in detail. But still I had to take off too much time from writing for doing other things. Months passed and I noticed I wouldn't be able to make the deadline. In January 2015 I re-planned the dates for book delivery, extending the deadlines for about two months.
Writing went well afterwards, I managed to get into some kind of flow more regularly but still I was taking too long. In March I extended for another month, planning on finishing the book in the end of April. In the end of March I noticed that even this plan would not work out and extended for another two months, finishing writing in the end of June 2015. I managed to meet this goal.
I logged all the time for writing in an excel sheet in 15 minute blocks. I became so obsessed with collecting the numbers that I also didn't work on the book if I had only five or ten minutes because then I couldn't log the time correctly.
Writing happened either in the morning before doing customer work or on dedicated days I spent at the library. It's good for me to have a dedicated work environment. Most of the days I felt really exhausted after working around six hours on the book. Writing is more challenging than normal programming work.
The Editorial Process
After I was done with writing a pdf was sent to some reviewers who had around three weeks to send their feedback. The result was really helpful with useful comments and suggestions. I incorporated some feedback for the final version, being extra careful to not add any errors because no technical reviewer would see those edits anymore.
This was also the time I did the index for the book, using the headlines as a guidance. Having a bad index can be really annoying for a technical book. This also took longer than you might think. I also had to redo the bibliography because I used the wrong notation. This basically meant typing all of the list again. I thought about some automation but in the end it was done by hand in a few hours.
After doing the edits I sent the manuscript to the publisher for final proofreading.
It was a bit of a surprise for me that the corrections arrived via mail and on paper. But it's the same with me – reading and correcting on paper works better.
After having added the corrections to the manuscript the book went off for typesetting which roughly took another two weeks. Afterwards I had to change some things regarding the code formatting and redo some of the images. The final version then went to print in mid November, the first big relief for me.
Holding the Book in the Hand
Before extending the deadlines I expected the book to be published in July 2015. When it was added to Amazon the initial release date stated September 2015, later October, then November and finally December 2015. I had a longer holiday booked for Christmas and I was really eager to hold the book in my hands before leaving.
Fortunately the first copy of the book was sent to me December 11 2015 and a few days later a box with all my author copies arrived.
Happy that my #Elasticsearch book is now available. A package full of past work :-) https://t.co/a4wWJpvmc0 pic.twitter.com/zhM1llaUbx
— Florian Hopf (@fhopf) 15. Dezember 2015
I can tell you, it's an excellent feeling holding the book in your hands after a year and a half of often stressful times.
The Website
Quite early I decided that it's a good idea to have a dedicated website for the book. I got me the domain elasticsearch-buch.de for marketing and for putting stuff on it that didn't make it in the book. Fortunately until today I didn't have to add an errata section :).
I created the website in the end of November 2015 using the static site generator Jekyll and a predefined template. I put up some articles and marketing texts and a list of all the resources in the book as clickable links. Even after the book was released I hadn't added all the content I was referring to in the book. There were some quite stressful last minute publish processes before leaving for a longer holiday.
After the release I also had the chance to publish a guest blog post about the book on the German elastic blog.
Compensation
It's no secret that this book won't make me rich. I don't know how many copies will be sold but in the end I expect the hourly rate for the writing to be in the low one digit € range. You might not be surprised that my consulting rates are a bit higher than that, but of course the book is good marketing for services. I received less requests for work than expected afterwards but having written the book I am more confident to request higher rates because I know the topic in depth.
But it's not only a money thing. I learned a lot while writing the book, going into far more detail I would have done otherwise, making sure that I really understood all the functionality I was describing.
Finally having written a book just feels good.
Statistics
Finally some numbers on the writing process.
| Number of commits in repo | 266 |
| First commit with chapter content | 16.06.2014 |
| Last commit before sending the book for publishing | 13.11.2015 |
| Number of time entries logged | 162 |
| Number of hours logged | 342 |
| Character numbers of all tex files | 495701 |
| Number of resources in bibliography | 199 |
I was a bit surprised to see that I had only logged 342 hours. So I could have done the book in around two months working 40 hours a week. Except that this wouldn't have been possible – there was a lot of conscious and unconscious thinking involved when I wasn't writing. Besides that it is nearly impossible for me to write for more than six hours a day.
On Versions
Elasticsearch is a very active project with lots of features added for new releases. I started writing the book using Elasticsearch 1.3.1, later switching to 1.4.x and then to the final version of the book 1.6. I was quite dedicated to ensure there are no mistakes in the example requests so after switching versions I had to make sure that all the requests were still working. This also means making sure the output of queries and the log output are still the same.
During the time of writing the rivers that were used to pull data in Elasticsearch were deprecated. I had already described the Twitter river to gather data from Twitter for use in the aggregations chapter. As switching this to another mechanism very likely meant that the document structure changed and I would have to redo all the aggregation examples I decided to stick with the river and explain how the same can be done using Logstash in a separate article on the website.
Another thing that happened in February 2015 was the rebranding of the company behind Elasticsearch to elastic. This meant that I had to redo all the links and all mentions of the company. This was more work than I would have thought in advance.
Two rather bad things happened with regards to versioning. In the chapter on centralized logging I describe Graylog as an alternative to the ELK stack. I used version 0.9.2 but in February 2015 Graylog 1.0 was released. As I had the chapter finished already including lots of screenshots and didn't have any time to spare I decided to stay with the version, stating it explicitly and just making sure that I didn't describe features that were gone in Graylog 1.0.
Even worse, after finishing the manuscript Elasticsearch 2.0 was released. It sounds like a big deal but lots of the changes were under the hood and it didn't change that much for the user. I wasn't able to redo everything for 2.0 but wanted to make sure that the book was also valid for the new version. I published a blog post on the website for the book describing the features and changes.
Generally I think it is very important to clearly state what version of software components you are using for writing the book. Otherwise readers might be confused when other versions don't work as described.
Conclusion
Writing a book can be extremely stressful but also is a very rewarding experience. Having a publisher with an editor means that you do have competent help when it comes to language and style (some publishers might be different and not focused on quality that much). The process of writing the book will never be paid from the sales alone. But it can be useful for marketing and it's a great way to learn about something you care about in depth.
The process is a classic example of Hofstadters law: It always takes longer than you expect, even when you take into account Hofstadter's Law. I wasn't able to spend that much time on writing as I was planning to, additionally reviewing and incorporating feedback took longer than I would have expected. Nevertheless I don't regret it – it was a good experience.