split mysqldump files with ruby

This is a ruby script I’ve used successfully to split up a mysqldump file that contained multiple databases.


#!/usr/local/bin/ruby

if (ARGV.count < 1)
  puts "give filename please."
  exit
end

if (ARGV.include? '--table')
    pattern = /^-- Table structure for table `(\w+)`$/
else
    pattern = /^-- Current Database: `(\w+)`$/
end

outfile = nil
File.new(ARGV[0], 'rb').each_line do |line|
  if line =~ pattern
    outfile.close if outfile
    outfile = File.new($1, 'w')
  end
  if outfile
    outfile.print(line)
  end
end

How to use OMW

So, this Android app that I wrote (see this post) is called On My Way.  It is easy for me to use (I wrote it) but as I was looking at it earlier today it occurred to me that it might not be as clear to everyone else.  Here are some things you should know if you want to use this app effectively.

  1. OMW will look back in time for two hours to find your next appointment.  For example, if you are supposed to be there at 9am then you can send the notification for this until 11am.  After 11am it will give up on that one and go look for the next event.  I chose two hours because that is a good time for ME, but if there are any requests I can change this time window or make it an option.
  2. If  you don’t want to send an email or text message, leave the corresponding field blank.  Otherwise you will be prompted to send by that method when you click the notification button.
  3. OMW does not detect failures of text messages or emails, so be sure your addresses and numbers are correct.
  4. You can change the message that is sent.  From the main screen (with the running guy) go into options and you will see the two templates, one for email and one for text.  Put whatever you want in there.  If you put %t in the message, it will be substituted for your arrival time.  That is the only substitution that is allowed currently.
  5. I have found the time estimates to be a few minutes off.  Of course, the phone cannot know about things like traffic and whether or not the driver will be speeding, so keep this in mind and adjust accordingly.
  6. Currently, OMW is most effective for drivers.  If you walk or bike to work, the time estimates will be little to no use to you.  Do not despair!  Edit your template in #4 and remove the %t.  The message will be more vague, but still informative to the recipient.  Maybe you could say something like “I’ve left and walking to work now, be there soon.”  Of course, if you’re walking to work you can probably afford the attention required to type out a message.

Well, that’s all I can think of now.  The way the app is programmed now it works perfectly for my usage scenario, so I don’t plan to make any changes.  If changes are requested and time permits, I will be happy to tweak the program.  You can leave comments on this blog, post issues at the redmine site, email me directly, post comments on the Market, etc. etc.

Cheers

On My Way

So, I’ve published my first Android application today.  This is a program called ‘On My Way’.  The idea is that if you are running a few minutes late for work, the phone should know this and notify your boss for you.  It is a smart phone, right?  The idea is simple enough, and the Android phones on the market have all of the necessary powers.  In a perfect execution, the phone wouldn’t even need your input, beyond the initial configuration.

When I sat down to design the exact requirements though, I had to compromise the vision a little bit.  Android might be smart, but its not really smart enough to tell the difference between a person being a little late for work and a sick day.  Besides, there could be other causes of insanity.  Better to have intervention from a Real Person before any email or SMS is sent.

Main screen

Main screen of On My Way

I programmed this app in three big bursts.  The first iteration was an ugly and barely working framework.  It had most of the business logic sorted out, resolving geo points from the GPS receiver into addresses, translating two addresses into a travel time, etc. but the interface was a disaster and the amount of scheduling options you had were, well, restricted.  Pretty much if you didn’t start at the same time every day Monday – Friday then it would only work on some of the days.

The second big push was to design a robust event description system.  I didn’t get as far with this as I would like, for example you can’t define an event that happens on the 15th of every month.  But for people with regular schedules there shouldn’t be any problems.  Each event can have only one start time, but if you need two different start times you can make two different events.  As many as you want, even, but I have a feeling that more than two or three would be hard to manage.  More complex scheduling options is a feature that this app could really use, but since I don’t need that feature it is not scheduled to be implemented.  If there is demand for it I can add it in an update…. An update to a premium version… HAHA!

new event screenshot

Configuring an event

The final project segment involved bug fixes and program polish.  I added the Beautiful artwork you see above, put those buttons on the event edit activity that let you pull phone numbers and email addresses from your contact list, added layouts to support landscape mode, squashed a few bugs and uploaded the apk to the market.

HURRAH.

not really.

I was sitting in a restaurant with my wife later that night (this night actually) and telling her all about my huge success, describing in detail the challenges I had mastering activity life cycles in Android.  Then, in my crowning crow, I let her watch as I downloaded the app from the market, opened it up, went to define the first event and crashed the app hard.  My FAVORITE words… Force Close.  How Embarrassing.  bleh.

So this is why On My Way is at version 1.1 on the first day of its release.  Weeee.

Anyway, if you are a user of this app and found this blog post looking for help with it, please shoot me an email at david@rubbersoft.com or you can file an issue at the project site for this program: http://rubbersoft.com/projects/show/omw.

Scan or click this link to install the app.

qr code

The link to the app

A fun screensaver

If your work place is anything like mine then you don’t dare to step away from your computer without locking it. The first time you do you are ruthlessly pranked by your cube neighbors. One popular prank is to flip the display upside down, or replace the desktop with a picture of the desktop, so the icons don’t work. Sometimes your “friendly” co-worker will hop on your email and send an email as you promising to take everyone to lunch.

This project is a trap for those who try these kind of pranks. A reverse prank the pranker type deal. What we do is we make a screensaver, password protected of course, that looks like you’ve left your desktop unlocked. Then when someone tries to mess with it, flash the screen a bunch of different colors and play an embarrassing alarm sound. haha!

Check out the code and submit issues at bitbucket: http://hg.rubbersoft.com/booby-trap-screensaver

Screenshots:

boobyconfig

Config page for boobytrap.scr

boobychoose

Choose the boobytrap screensaver

RubberStumble: or How to Bounce around the Web

RubberStumble is an extension I wrote for Google Chrome that was inspired by my burning need to Stumble.  I quickly fell in love with Chrome when I discovered it, but not having that stumble button was a deal breaker in the early versions of the browser (version 1 and 2).  I learned something about my habits in those days… Whenever I finished with the webpage I was on, I moved my mouse without thinking to click Stumble.  Then there was a brief moment of confusion when the Stumble button wasn’t there, and I would sigh.  Well, as you can imagine, a man can only put up with that for so long.  I didn’t plan on changing my habits, so I had to change my browser.  Back to FF.

This may seem quite strange if you’re not familiar with StumbleUpon.  Let me describe it briefly.  It started as an extension for FireFox, a toolbar that has a green button on it, the Stumble button.  When you click the button, it takes you to a random page on the internet.  The pages in the stumble database are submitted by users, and divided into categories.  You register your account at StumbleUpon.com, and choose your categories (like computer, math, and cats).

The genius of StumbleUpon is that it will take you to places you never would have thought to go before.  Sure, if you know what you’re looking for on the internet, then you can google for it and probably find it.  But there is a ton of funny and interesting content out there that you would never think to google for.  For example, here is a blog post I just stumbled, a list of 10 places you can never go.  An interesting read that taught me things I would never have googled for.  Here’s another: Bear and Fox comics.  I’ve never heard of ‘em before.  This comic features a product called “Crybaby lemonade”.  Ok, maybe I’m not quite getting that joke, but whatever.  It took me 20 seconds to click stumble, read that comic, and then click stumble again.  YAY!.  So you can see why I could not participate in any non-stumbling browser.

A few months later I was inspired to revisit Chrome when I saw that they had implemented the beginnings of an Extension framework.  I was excited to see what was out there, and I was confident I could find an extension for stumbling.  So, I read the documentation for the Extension framework, and started poking at the Firefox StumbleUpon extension to see what secrets it held.  First I looked at the files of the extension that were installed on my computer.  On windows Xp the files can be found here:

C:\Documents and Settings\<USERNAME>\Application Data\Mozilla\Firefox\Profiles\<somestring>.default\extensions\{AE93811A-5C9A-4d34-8462-F7B864FC4696}

This folder has a compiled jar file and a couple of plain text .js files.  Reading through the file named StumbleupongService.js I discovered the URIs of some other scripts, most importantly: chrome://stumbleupon/content/datastore.js.  Put this URI into firefox (if you have the extension installed, of course) and you get to download a javascript file just under 3k lines.  Now, this file doesn’t reveal all of the secrets of StumbleUpon, but pretty much everything that has to be done browser side is done here (as far as i can tell).  From this file I learned how the password is encrypted to be sent over the wire to the stumbleupon servers.

Next, I used a tool called Fiddler Proxy to get a transcript of a StumbleUpon session, to recreate it with my own extension.  It basically goes like this:

  1. Browser POSTS a login request to StumbleUpon.
  2. SU replies with the results.
  3. Browser POSTS a request for recommended pages for the user.
  4. Browser sends back a list of 3 websites with various metadata on each.
  5. Browser navigates to the first page.
  6. Browser notifies SU that it went to that page.

After step 6, when the button is clicked again then the extension goes either to step 5 or step 3, depending on whether the last batch of recommended pages is exhausted or not.

I had the most difficult time recreating step 1.  See, the password that the user types in is not sent to SU, rather they send the SHA1 hash of that password.  But whenever I tried to build a http request that matched the one I was seeing, I couldn’t get my input to create the same hash.  Turned out that the hash was correct, but the results were encoded differently by mozilla and the library I was using.  My library (courtesy of webtoolkit.info) returned the results encoded as hex, and the mozilla library. returned them encoded in base64.  Once I changed sha1.js to return base64 strings, things started clicking.

I was quickly able to get SU to send me a list of recommendations.  For a while I tried to avoid step #6, but eventually I had to implement it because SU kept sending duplicate pages.  Working with the google chrome browser was pretty easy, the documentation was clear and because the extension framework was small it was easy to take in all of the features.

I originally built the UI to the extension as a browser tool strip, this was the way it was done in the very early stages of the chrome extension API  (before the gallery opened).  The tool strip solution was not very good, the way it worked was that there would be an additional toolbar just for extension controls at the bottom of the browser window.  It’s pretty inconvenient to click on something at the very bottom of the screen though, if you have to go there frequently.  But it worked and I was happy.  Eventually I had to update the interface to use Browser Actions instead of Toolstrip buttons, but this was easy and a much nicer solution.

The official SU extension for chrome was released a few months later, when the gallery opened.  I haven’t tried it yet, but I’m sure its nice.  It is guaranteed to have more features than RubberStumble, like the ability to vote pages up or down, or to write reviews on pages, or to change what category or keyword you are stumbling on.  RubberStumble will only do one thing… stumble.  If I can come up with a way to add more features without taking up more screen space, or without requiring more clicks to get to stumble, I may add some.

RubberStumble can be installed from the extension gallery.  The redmine project page for RubberStumble is public, anyone can post bugs or request features or post on the forum over there.  If you want the source code, you can check it out here by SVN.

What about DCLockoutStatus?

There are a few projects hosted over at the redmine site for Rubbersoft that don’t get much attention.  Most of these are programs that come about when I have an itch for a program and can’t find a download that scratches it.  Or sometimes there is a program out there that I like, but it doesn’t work exactly the way I think it should, so I re-write it and include the feature I wish it had.  An example of this is DCLockoutStatus.exe.

dclockoutstatus_main

Main window of DCLockoutStatus

Some may recognize this user interface, it is basically the same as Microsofts own lockoutstatus.exe.  This tool from MS was one that I relied on heavily when I worked in a domain with sites across the continent.

Let me back up a moment and briefly describe the problem that these programs try to resolve, for those not familiar.  Active Directory (AD) is the windows program that manages users in a enterprise windows network.  Users are defined in the AD database which is hosted on special servers called Domain Controllers (DC).  Whenever a user wants to log onto one of the workstations in the domain, that computer talks to a DC over the network to see if the password is correct.  This works pretty good as long as the user is actually close to a DC, like in the same building.  If a user is far from a DC, like in another state, then there is a problem: the delay could be significant.

So Active Directory has a provision for putting Domain Controllers in different physical locations.  Each location is called a site.  Then, when the user logs onto his computer, the computer just needs to talk to the DC at the site he is at to find out if it’s ok to let that person log on.  This speeds up the login process, but introduces a different problem, replication.  All of the Domain Controllers in the domain need to agree on what a persons password is, whether or not that account is locked out, etc.  So they need to talk to each other and send updates to each other when something changes.  But not too much, you see.  The Active Directory database is large, and fields are changing in that database frequently.  If a replication occurred instantly when a change happened, the network may be overwhelmed, or more important network traffic (like me trying to find the score of my fantasy football team) may be delayed.  Cognizant of this danger, MS implemented a replication scheme where the most important data (like a users password) is replicated instantly, less important data is replicated on scheduled basis, and trivial data is not replicated at all.

The question of whether or not a user is locked out of his/her account falls into that second category.  As a result, there are times when a user is locked out at one site, but not another.  This causes confusion occasionally.  Example: user calls helpdesk to say they are locked out.  They are at site B.  The helpdesk is at site A.  The helpdesk agent opens his AD program:  Active Directory Users and Computers, or ADUC (pronounced A-duck).  ADUC  connects automatically to the DC at the agents site.  They pull up the users account, and it does not show to be locked out.  You can imagine what happens next.  Confusion, hysteria, and sometimes cursing.

  • Helpdesk: “Is your caps lock on?”
  • User: “no.”
  • Helpdesk: “Try pushing the caps lock key and try again.”
  • <etc>.

Microsoft’s program, lockoutstatus.exe, is meant to relieve some confusion caused by this situation.  It will connect directly to each domain controller in the domain, and tell you if the user is locked out on that server or not.  The problem with lockoutstatus.exe is this: it will not actually let you unlock the account or reset the password from that interface.  Oh sure, it has the controls to do so, but every time I’ve tried to use them, I got an error message.  This wasn’t a one-off either, it’s happened in every domain I’ve tried it, across three organizations.  At the third org, I was frustrated and started looking at the .net API’s for Directory Management.

Actually writing the replacement program took under a week.  The user interface was already designed for me (thanks MS!).  There are easy .NET calls for getting a list of domain controllers and sending commands to them.  Writing DCLockoutStatus was just a matter of gluing a list of domain controllers to a list view and adding a context menu, and a few buttons.

I distributed this to my team, to warm reception.  We found some unexpected benefits as we used the app.  For one, it is quicker to launch and search for users than ADUC.  For two, it will show the age of the password easily, and the time of the last bad password attempt, which ADUC does not show.  It will show how many bad passwords have been attempted by the user.  All in all, it is an effective tool for password and lockoutstatus management in a domain.  It won’t replace ADUC… you can’t create users, or see email addresses, or do any of the thousand other tasks that ADUC can do.  But when it comes to passwords, I have a hotkey that opens DCLockoutStatus.

There is room for many more features in this program.  Some things I’d like to add include support for remote domains other than the login domain, a desktop gadget interface, and the ability to monitor a particular account and pop a notification if it becomes locked out, and information about what device the lockout came from.  I’ll be revisiting this program when I finish up some other projects.  If you’re interested in DCLockoutStatus you can see the source code at the redmine project page.  I haven’t posted an installer for it, but if someone wants one I could put it together in a few minutes or so.

Special chrome pages (4.0)

note: many of the pages in versions 3.0 and earlier have been removed or renamed. about:stats for example. The 3.0 pages are not discussed here.

All browsers use special URLs to display troubleshooting and debugging information to advanced users. Google Chrome has a wealth of these. Some are useful, some are interesting, some not very useful for the average user. A few of these are discoverable by clicking on some items in the menus. Others are hidden, and not intended for public consumption. Many of these I discovered from the Chromium source code. These pages are all working on version 4.0.245.0. In earlier versions of chrome, many of the pages either don’t exist, or have a different URL. The sample I have below is not exhaustive, there are URLs that describe resources that you could technically access, but only shows an image.

chrome://extensions

shows installed extensions
extensions

chrome://net-internals

net-internals

chrome://newtab

knowing the url for this page is helpful if you want to open two or more of them when chrome starts

chrome://net-internals/view-cache

Also known as about:cache
A useful page for troubleshooting odd problems with web page display.

chrome://net-internals/view-cache/*

where * is the page
To see the browser cache of an individual page.

chrome://downloads/

Download manager

chrome://history/

History manager

chrome://syncresources/mergeandsync

sync

chrome://inspector/inspector.html

opens the DOM inspector

about:histograms

histograms

about:plugins

plugins

about:credits

credits

about:memory

memory

about:shorthang

This will cause the current tab to hang briefly.

about:dns

dns

about:version

shows the version.

about:sync

aboutsync

about:crash

going to this address will crash the current tab.

about:hang

Permanently hang the current tab.

Bonus link: http://src.chromium.org/viewvc/chrome/trunk/src/webkit/glue/webkit_glue.cc?r1=8764&r2=8763&pathrev=8764#l417
Check out line 419. hah!

CustomNewTabpage.crx

This is an extension for google chrome (4.0+) that will allow you to put in any url that you want to be the newtab page.  To set a custom new tab page, open the options page for this extension and you will go to the page where you can change the newtab URL. Install the extension by clicking on the attachment at the bottom of the post, or this link:  CustomNewTabPage. If you want to revert to the default newtab page, with the thumbnails, then uninstall the extension.

The code for this extension is rather simple:

option.html

<script type="text/javascript">// <![CDATA[
    function update_homepage() {
        var homepage = document.getElementById("homepage").value;
        var re = /^(https?|file|ftp):\/\//;
        var m = re.exec(homepage);
        if (!m) {
            homepage = "http://" + homepage;
        }
        localStorage.homepage = homepage;
        console.log("Set home page to " + homepage);
        var o = document.getElementById("output")
        o.innerHTML = "Set home page to " + homepage;
        o.style = "";
        return false;
    }
    function init() {
        if (localStorage.homepage) {
            document.getElementById("homepage").value = localStorage.homepage
        }
    }
// ]]></script>
<h1>Enter what you want to be the newtab page</h1>
<input id="homepage" type="text" /><button onclick="update_homepage();">Update</button>

redirect.html

<script type="text/javascript">// <![CDATA[
    var homepage;
    if (localStorage.homepage) {
        homepage = localStorage.homepage;
    } else {
        homepage = "chrome-extension://cpknocklipbllidbjjngmcnmadhhpbgo/option.html"
    }
    document.write("<meta http-equiv='refresh' content='0;URL=" + homepage + "'")
// ]]></script>

manifest.json

{
"name": "CustomHomepage",
"version": "1.0",
"description": "Enables setting a custom home page",
"chrome_url_overrides": {
"newtab": "redirect.html"
}
}

Install with this link:CustomNewTabPage

RedVisitedLinks.crx

An extension that I wrote that will change the color of visited link in Chrome to Red. There are two files in this extension:
manifest.json:

{
"name": "RedVisitedLinks",
"version": "1.0",
"description": "Make visited links red",
"content_scripts" : [
{
"matches" : ["http://*/*", "https://*/*", "ftp://*/*"],
"css" : ["redlinks.css"]
}
]
}

and redlinks.css:

a:visited { color: red; }

TaDa!

Install by clicking the link below. Requires version 4.0+ of chrome.

RedVisitedLinks

Introducing AutoPowerShell and this Blog

So, I’ve made a few posts already, eager to begin as I was. Reviewing the content so far, I have come to realize that this beginning was not well planned. Allow me a minute or so to Introduce myself and my product.

My name is David Warburton and I’m going to write about scripting, programming, and AutoPowerShell in this blog. (Also, expect pictures of my children occasionally). AutoPowerShell is a program I wrote, a system tray host for Powershell. You can read about it in the first post.

AutoPowerShell launcher

AutoPowerShell launcher


This is the main launcher window for AutoPowerShell. The actions you define will appear as buttons in this window. Just about any powershell script you would normally run in the console will work in AutoPowerShell. There are a few caveats, some of the cmdlets that deal directly with the powershell console window, like write-host, may not work. You can still write output… it will appear in a popup window when the action is executing.

The idea behind AutoPowerShell is Windows Automation. Using powershell scripts to control the keyboard and windows is not practical, because when the script is launched from the default powershell host (the console) the window that has focus will be the console itself. Also, the default host is pretty bad for setting up native hotkeys. Both of these problems are eliminated with AutoPowerShell.

In the coming months, I’ll post examples of what can be done with AutoPowerShell. Some of these scripts would work just as well with the default host, some of these are better suited for AutoPowerShell, some of these are only possible with AutoPowerShell.

Also, as a companion to AutoPowerShell, I plan to develop a module that is designed for windows automation in the context of AutoPowerShell. This will probably be a culling and combination of the various valuable community contributions that are out there, such as WASP and PSCX.

AutoPowerShell install file can be downloaded over at http://autopowershell.com. Check the Files tab for the latest released msi file. You can also file bug reports, feature requests, or make comments at that site. For any support or questions, you can write to me at .

Now with posts!