Personal System Iterate: Weekly Review Init
04/07/15
So, over the past week, in my diary area, I have been working on a script that interacts with my blogpost writings and... generates formatted html that will link to all of my posts since my last weekly review.
It's not a huge, amazing thing, but it is the building blocks to various powerful things, and it is also composed of smaller building blocks that helped me arrive there. It was also a successful breakdown of a project that would take me more than a few bus-rides such that I knew I was making progress each bus ride. Almost all development of this script was done during bus-rides. In fact, this post is right now being written on the bus. Basically, the bus is great for productivity, but that's maybe a post for later.
So, the idea was that I would write about what I was doing while I was doing it. This was valuable for several reasons:
- Rubber-duck debugging
- Detailing what I wanted to make
- Detailing what I was currently making
- Connecting the two together
It is now also valuable in the sense that as it turns out I like the posts that came out of this documentation-process, and now I have content that I can condense into a proper post on some javascript node scripting that I have accomplished.
Background
This site that you're reading is currently a bunch of strung-together html files. There's nothing much there beyond that. Being a programmer by trade, that feel kinda unacceptable to me. It's fine as a beginning sort of prototype, but I can do better, and I think it being better would allow me to do better stuff here and elsewhere.
I tried to fix this before, back when I was playing with python server tools, but I made the mistake of trying to do it all at once, getting lost and/or stuck, and giving up.
I've come to realize that breaking things down wasn't something I was ever particularly great at. That's something that school does for you a lot of the time, and being great at school thus meant I wasn't necessarily great at that. I'm getting there, however. It actually isn't that hard, I realize. It's a matter of looking at the big-picture, identifying a necessary piece, prototyping that piece in a hard-coded environment, verifying the validity and value of the piece, and iterating again from the beginning from there.
So, here, the big picture is that I want my site running on node, and for it to be as DRY as possible. Now, to a certain extent, I'm doing that for its own sake, because I think I can do better, but I believe learning to do this will open up pathways to ever-more-interesting adventures, and so I'm on this quest for the sake of emergent future quests, as well.
Which, as an aside, is totally something that people do in RPGs, so maybe I'm also gamifying my self-growth.
Anyways, as far as DRYness goes, I've made a findReplace script for the sake of grabbing all the places I have a browseBar (that thing that links to various bits of my site or things relating to me), and editing them to all more-or-less be uniform.
And now, for this post, I will elaborate on the development of a script that reads through my diary and my blog, and grabs all the posts since my last diary post titled "Weekly Review" (more info on what that is here).
Now, how exactly does this do either of my big-picture things? Well, it gets me more familiar with node, interacting with html server-side, manipulating it, and generating other html. I mean, the only thing really missing is actually serving the code to a browser. Also, manipulating the html is a step itself towards DRYness, as I can manipulate it into a DRY form at some point, and make sure that interactions all still work as I desire.
All right, without further ado, the development process and the code.
The Process
Going in, I had no idea how to even really start. I mean, at work I went line-by-line through my stuff in python, and just hacked something together based on various html properties, and how I organized it there-of.
This was obviously not sutiable, and I knew there'd be a better tools available to me via the wide-open Internet and the fact that node is made for this kinda stuff, so there best be tools for such interaction.
Finding the Tools and Testing the Waters
I didn't know how to go about what I wanted to do, but I had heard of an html parsing library called cheerio at the TucsonJS meetup. It has a jQuery API, and I figured this would be a great tool for my exact goals.
My first bit of code was basically a copy of readme example:
/*jslint node: true */
var cheerio = require('cheerio'),
$ = cheerio.load('<h2 class="title">Hello world</h2>');
And thus began my growing familiarity with cheerio.
Loading in an HTML File
This was an obvious next-step. I needed to actually interact with my files, so I needed to feed it into cheerio somehow.
/*jslint node: true */
(function () {
"use strict";
var args = process.argv.slice(2),
fs = require('fs'),
cheerio = require('cheerio'),
$,
usage;
usage = function () {
console.log("Need a single input HTML file to read.");
};
if (args.length !== 1) {
return usage();
}
fs.readFile(args[0], "utf-8", function (err, data) {
if (err) {
return usage();
}
$ = cheerio.load(data);
console.log($.html());
});
}());
I made it so I could feed in any file as a command-line argument, and basically just print the file out. But that I got that from a $.html() implies I can get anything that $ is willing to give me.
For instance:
fs.readFile(args[0], "utf-8", function (err, data) {
if (err) {
return usage();
}
$ = cheerio.load(data);
console.log($('.blogpost').length);
});
This grabs all my blogposts (marked by the class="blogpost" on all of my blogpost divs), and prints out their length. When I first made this, the ouput was:
$ node blogpost.js public/blog/index.html
239
This was verified correct at the time, and already showed me that I could get a variety of value out of using this cheerio library.
Working Towards Use
All right, it was around here I had some trouble and really needed to read the documentation. All I wanted to do was printout the title and the date of each blogpost since my last weekly review, but I was unsure of how to only grab the h1 and h2 text within each blogpost div (h1 contains the title and h2 the date).
I also didn't know how to remove whitespace. I ended up hacking things together.
$ = cheerio.load(data, {
normalizeWhitespace: true
});
blogposts = [];
$('.blogpost').each(function (index) {
var that = this,
validPost = function () {
if (date && $(that).children("h2").text().search(date) === -1) {
return false;
}
if (!date) {
if ($(that).children("h1").text().replace(/\ /g,'') === "WeeklyReview") {
date = $(that).children("h2").text();
}
}
return true;
};
if (validPost()) {
return blogposts[index] = $(this).children("h1,h2").text();
}
return false;
});
console.log(blogposts.join('\n'));
At the time the code output:
$ node blogpost.js ../public/diary/index.html
Plan Document 04/01/15
Log 04/01/15
Personal System Iterate: Weekly Review Init 04/01/15
Plan Document 04/01/15
Log 03/31/15
End of Day Review 03/31/15
End of Day Routine 03/31/15
Personal System Iterate: Weekly Review Init 03/31/15
Plan Document 03/31/15
Plan Document 03/30/15
End of Day Review 03/30/15
Weekly Review 03/30/15
UA Game Dev Club Meeting 03/30/15
Plan Document 03/30/15
Again, at the time this was the correct answer. But it was a struggle to get this together, and now that I had it, some organization was necessary.
Clean Up
Hacking something that works was good, trying to go any further without cleaning up after myself would have been a mistake, and I successfully avoided that very mistake. And so I cleaned up a bit:
$('.blogpost').each(function (index) {
var blogpost = this,
whitespace = /\ /g,
valid,
date,
title;
title = function (blogpost) {
return $(blogpost).children('h1').text();
};
date = function (blogpost) {
return $(blogpost).children('h2').text();
};
valid = function () {
if (earliestDate && date(blogpost).search(earliestDate) === -1) {
return false;
}
if (!earliestDate) {
if (title(blogpost).replace(whitespace, '') === 'WeeklyReview') {
earliestDate = date(blogpost);
}
}
return true;
};
if (valid()) {
blogposts[index] = $(blogpost).children('h1,h2').text();
return;
}
return false;
});
And the code feels more readable now, no? Still hacking away the whitespace with a regex statement, but I don't know better yet. But seeing the code made me think of how I may go about also grabbing the link to each blogpost, a necessary part of linking to each blog post since the last weekly review.
After a few false starts, I found what I was looking for and modified the necessary bits of code:
if (valid()) {
blogposts[index] = (
$(blogpost).children('h1,h2').text()
+ $(blogpost).find('a').attr('href')
);
return;
}
Which spits out:
$ node blogpost.js ../public/diary/index.html
Plan Document 03/30/15 plan_document_150330.html
UA Game Dev Club Meeting 03/30/15 gamedevclub_150330.html
Weekly Review 03/30/15 weekly_review_150330.html
End of Day Review 03/30/15 end_of_day_review_150330.html
Plan Document 03/30/15 log_150330.html
Plan Document 03/31/15 plan_document_150331.html
Personal System Iterate: Weekly Review Init 03/31/15 personal_system_iterate_weekly_review_init_150331.html
End of Day Routine 03/31/15 end_of_day_routine_150331.html
End of Day Review 03/31/15 end_of_day_review_150331.html
Log 03/31/15 log_150331.html
Plan Document 04/01/15 plan_document_150401.html
Personal System Iterate: Weekly Review Init 04/01/15 personal_system_iterate_weekly_review_init_150401.html
Log 04/01/15 log_150401.html
Plan Document 04/02/15 plan_document_150402.html
Personal System Iterate: Weekly Review Init 04/02/15 personal_system_iterate_weekly_review_init_150402.html
Log 04/01/15 log_150402.html
Plan Document 04/03/15 plan_document_150403.html
And at this point I figured I knew everything I needed to really make what I wanted. Which meant one thing.
Starting Over
I had made my prototype to prove I could do what I wanted. It was built out of exploration and that structure was not necessarily ideal, just emergent. Starting over lets me rethink the structure of my code from a design focus, not an exploratory one.
/*jslint node: true */
(function () {
'use strict';
var fs = require('fs'),
$ = require('cheerio'),
before,
linkText,
last,
date,
title;
date = function (blogpost) {
return blogpost.children('h2').text().trim();
};
title = function (blogpost) {
return blogpost.children('h1').text().trim();
};
linkText = function (postList) {
return function () {
var link,
blogpost = $(this);
link = function (blogpost) {
var ahref = blogpost.find('h1 a').attr('href');
if (ahref) {
return ahref.trim();
}
return '';
};
postList.push(
'<h3>\n <a href="'
+ link(blogpost)
+ '">\n '
+ title(blogpost)
+ '('
+ date(blogpost)
+ ')'
+ '</a>\n</h3>'
);
};
};
before = function (givenDate) {
var compareDates,
givenMonthDayYear;
givenMonthDayYear = givenDate.split('/');
compareDates = function (date) {
var monthDayYear = date.split('/');
if (monthDayYear[2] !== givenMonthDayYear[2]) {
// Compare years
return monthDayYear[2] < givenMonthDayYear[2];
}
if (monthDayYear[0] !== givenMonthDayYear[0]) {
// Compare months
return monthDayYear[0] < givenMonthDayYear[0];
}
if (monthDayYear[1] !== givenMonthDayYear[1]) {
// Compare days
return monthDayYear[1] < givenMonthDayYear[1];
}
// Default compare strings
return date < givenDate;
};
return function () {
var blogpost = $(this);
return compareDates(date(blogpost));
};
};
last = function (blogposts) {
return date($(blogposts.get(0)));
};
fs.readFile('public/diary/index.html', 'utf-8', function (err, data) {
var diaryPost = [],
weeklyReviews;
if (err) {
return;
}
weeklyReviews = $('.blogpost', data)
.filter(function () {
var blogpost = $(this);
return title(blogpost) === 'Weekly Review';
});
// Just this once, I want the weekly review before last
// This is a quick hack that filters out the most-recent weekly review
weeklyReviews = $(weeklyReviews).slice(1);
$('.blogpost', data)
.not(before(last(weeklyReviews)))
.each(linkText(diaryPost));
fs.readFile('public/blog/index.html', 'utf-8', function (err, data) {
if (err) {
return;
}
$('.blogpost', data)
.not(before(last(weeklyReviews)))
.each(linkText(diaryPost));
console.log(diaryPost.reverse().join('\n'));
});
});
}());
And here is the code I currently use. Not really complete, but it has the major functionality that I desire. I finally discovered the trim function, the built-in whitespace remover for javascript. I actually really like how a lot of the code reads. In particular:
$('.blogpost', data)
.not(before(last(weeklyReviews)))
.each(linkText(diaryPost));
Writes exactly what it does.
Now, looking at it again, I don't like a lot either, but they're all contained in stand-alone functions that I can clean-up without having to touch other code. For example, the before function is a little convoluted and could be executed better. How I call the code, however, I like plenty, and it makes those pieces more-or-less properly self-documenting.
Looking Towards the Future
I like the script I got out of this experience, and I feel I've learned a lot that will help in the development of future scripts that will slowly transform my site to my liking.
I think the next thing I'll do is make a script that finds all those blogposts that didn't get their own links early on, and generate the html necessary such that they have a link, and that link leads to the proper post.
I think with my current understanding and capabilities, I should be able to pump out that script quickly.
We'll see.