Wouldn’t you love to have a crime map like this for your New York community?

The momentum toward open government data in the United States is buoyed by the following principle: government must be transparent. Governments collect huge amounts of data. They should make that data freely and easily accessible to citizens on whose behalf they collect the data in the first place.

The emergence of inexpensive technologies that enable open data initiatives is accelerating that momentum. Just as I can upload my personal data (documents, photographs, videos, music, etc.) to a cloud service and then retrieve it when and where I want to, governments can now upload their data sets to the cloud, allowing their citizens to retrieve that data and analyze it in a way that makes sense to them.

What community wouldn’t love to have access to a crime map like the one above, for example? Clever technology renders an overwhelmingly large data set easy to use by any citizen who can interact with and drill into the map to see extraordinary detail on recent crime in San Francisco’s neighborhoods.

Inexpensive, cloud-based, off-the-shelf solutions

There are many open data cloud services emerging. For example, Socrata serves the Unites States, numerous states and large cities, among them New York City. By taking advantage of the economies of scale and pervasive access of cloud computing, Socrata can increase citizen access to government data and online services and at the same time lower the government’s costs of delivering that information.

By using cloud-based services like Socrata’s, New York State can eliminate the need for high-cost, custom application development or updates to legacy systems. Because there is no need for a lengthy multi-million dollar information technology project, departments simply begin to upload their .csv and .txt files, spreadsheets and databases to the open data portal service. Data sets immediately become available. Transparency is achieved. Citizens are informed and empowered. Job done.

How things could be in New York State

Let’s say I want to know the average expenditure per student by the state’s school districts. In Oregon, I simply go to the open data portal and search “school district expenditures.” I am immediately presented with a data table that I can browse, search, sort and filter. In addition, I can view the same data as a bubble chart created and saved by another user. Finally, I can embed the data table in this blog!

In New York State, there is no open data portal. So I search “school district expenditures” in Google, which returns 39 million results. After the first 20 hits, I have not seen a link to a data set but only articles and news reports. Frustrated, I go to the New York State Education Department website and search. There, the same query returns dozens of articles, legal briefs and technical reports but no data table that I can browse, search, sort and filter. As Oregon proves, it does not have to be this way.

As easy as getting a weather forecast

Imagine that finding the New York State data you need were as easy as accessing the weather on your smart phone. In states that have implemented open data cloud services, it is often that simple — and it should be that easy in New York State. Further, open data cloud services create robust markets for apps that improve your quality of life and make government more transparent.

Open data cloud services are disruptive, game-changing technologies that fundamentally redefine the relationship between citizens and government data. They make access to data easy, efficient, and inexpensive. In fact, the implementation of an open data platform in New York State will make the old “technology” for doing so — the arduous, time consuming and often expensive Freedom of Information Law (FOIL) request — obsolete in many cases. Indeed, we in New York State can have sunlight without the pain — just as they do in Oregon, Oklahoma, Utah and Illinois.

The bottom line

New York State should immediately implement open data cloud services and make its data sets easily available in a single, statewide portal with tools and facilities that allow citizens to browse and analyze that data easily. If other states and cities, including Mayor Bloomberg and New York City, can implement open data, then why not Governor Cuomo and New York State?

The Capitol Pressroom can be heard on public radio throughout upstate New York on stations including Buffalo (WBFO), Rochester (WXXI), Syracuse (WRVO), Binghamton (WSKG) & Albany (WVCR). You can also listen by live stream or podcast.

Open data initiatives allow citizens to browse open data portals in order to find, download and use machine-readable data sets generated by the government. The United States, the United Kingdom, Canada and even Kenya have open data portals. States such as Oregon, Utah and Illinois and cities such as Baltimore, Chicago and New York do as well.

Free and open access to data:

• Spurs innovation and economic development
• Increases government transparency and accountability
• Reduces waste and fraud
• Improves our quality of life
• Strengthens our democracy.

No one wants the user to have a clunky experience and no one wants to maintain superfluous plugins or parallel mobile sites. Fortunately, there is now a simple and elegant solution: responsive web design.

Responsive Web design is the approach that suggests that design and development should respond to the user’s behavior and environment based on screen size, platform and orientation…As the user switches from their laptop to iPad, the website should automatically switch to accommodate for resolution, image size and scripting abilities. In other words, the website should have the technology to automatically respond to the user’s preferences. (From Smashing Magazine)

While reading this post, you are experiencing the benefits of responsive web design. If you are on a desktop computer, shrink your web browser window. Notice that as the browser window narrows, the web site automatically re-renders appropriately. The banner shrinks, the menu bar starts to stack itself, media resizes, the side bar repositions itself below the page, the footers stack, and, if the page has multiple columns, they stack too. If you are on mobile device such as a smart phone or tablet, you are already experiencing the rendering flexibility I have just described. If you are not on a mobile device, switch to one to see what I mean.

To see how well or poorly your website renders on a variety of devices: go to Screenfly, enter your website’s URL, select the icon at the bottom of the page (i.e., Desktop, Tablet, Mobile, Television) and then select a device (e.g. Mobile –> iPhone 3).

There is compelling reason to evolve your organization’s web site to responsive design now. According to Ethan Mercotte, “Mobile browsing is expected to outpace desktop-based access within three to five years. Two of the three dominant video game consoles have web browsers…We’re designing for mice and keyboards, for T9 keypads, for handheld game controllers, for touch interfaces. In short, we’re faced with a greater number of devices, input modes, and browsers than ever before.”

Responsive web design brilliantly manages current and emerging innovations in mobile browsing — elegantly for your users and effortlessly for you.

We at Prism espouse agile app development. We like simple, useful software that is frequently updated. No surprise, then, that we have embraced responsive web design.

If you are interested in responsive design and would like more information, please contact us.

These portals allow the public to find, download and use machine-readable data sets generated by the government. Indeed, the US’s data.gov now provides access to thousands of data sets.

New York City is also emerging as a leader. Mayor Bloomberg’s NYC BigApps 3.0 competition is currently underway. The contest offers $50,000 in cash and other prizes for the best new apps that use more than 800 New York City data sets to make government more transparent and improve “the daily lives of New York City residents and visitors.”

Open data initiatives have produced a myriad of innovations. We can locate superfund sites or visualize the health of our communities. New York City commuters access and update real-time subway, bus and driving conditions. San Francisco drivers locate open parking spaces and their cost. Baltimore’s 311 mobile app enables citizens to receive or share crisis data during natural disasters and to report the locations of pot holes, graffiti or broken streetlights. In the United Kingdom, the government is publishing data including every contract and item of government spending over £25,000.

In the meantime, New York State lags miserably. Interested citizens, researchers and entrepreneurs who seek financial data from the Comptroller’s Office or student performance data from the State Education Department, for example, will find themselves confronted with machine-inaccessible formats, such as PDF. Because the data provided is not machine-readable or linked, it has significantly diminished utility.

In New York State, access to data is limited, citizens are in the dark, researchers are frustrated and app developers go elsewhere. There is an alternative.

If New York City can do it, why not New York State?

Free and open access to data:

• Spurs innovation and economic development
• Increases government transparency and accountability
• Reduces waste and fraud
• Improves our quality of life
• Strengthens our democracy.

Governor Cuomo, there is a compelling ethical, civic and economic case for New York State to provide access to high-value, machine-readable public data sets.

Please, free the data.

Respectfully,

Sean Brady, President, Prism Decision Systems, LLC
John Sipple, Associate Professor, Cornell University & Director, New York State Center for Rural Schools

The concept is simple — by tracking factors that research shows are key indicators of dropout risk — credits, commencement exams, attendance, discipline referrals and out-of-school suspensions — school leaders can keep a bird’s-eye view of each cohort of students and more easily identify those who need extra support to get to graduation day.

Of course, once schools leaders have a better idea of who is most at risk, there is simply no substitute for good, old-fashioned compassion — and good, old-fashioned elbow grease.

Indeed, with a smart tool like CohortTracker™ reducing the effort required for easy and accurate early identification, there is now more time and energy left for that “good, old fashioned elbow grease.” Educators, roll up your sleeves!

For more information see:

• Screen cast introduces CohortTracker™
• CohortTracker™ early warning system

January 27, 2012:

Binghamton, NY – The Greater Southern Tier Board of Cooperative Education Services (GST BOCES) has implemented CohortTracker™ in fourteen school districts in its region. CohortTracker™ is an early warning web application that allows schools to monitor students’ progress towards graduation. Educators can identify students getting off track and intervene before it is too late. The app is licensed by Prism Decision Systems, LLC.

According to Steve Manning, Manager of Computer Services for GST BOCES, “We are very pleased to have gone live with CohortTracker™. The app is simple, user friendly and extremely effective. School principals and guidance counselors will find this early warning system to be a valuable new tool for tracking and managing their students to successful, on-time graduation.”

GST BOCES, which has exclusive license to CohortTracker™ in New York State, will now turn its attention toward statewide rollout. “We hope to have the app available to all New York State BOCES Regional Information Centers before the end of this school year,” Manning said.

“CohortTracker™ is designed to be a general-purpose student achievement tracking system that can be tailored to meet the specific needs of any state, regional or local education department,” said Prism president, Sean Brady.

Prism embraces an agile approach and focuses on developing apps that

• Are lightweight, solve a single, urgent problem, and are not burdened by unnecessary features or complexity.
• Deliver focused, actionable, real-time information to educational decision-makers at all levels—in a matter of weeks, not months or years.

“We believe agile apps are the future of data-driven decision-making in education,” Brady said.

For more information see these additional posts:

• Screen cast introduces CohortTracker™
• CohortTracker™ early warning system

About Prism Decision Systems, LLC: Prism Decision Systems provides dynamic group processes and state-of-the-art decision support to clients in the education, health care, government, finance, news and entertainment, energy, telecom, technology and transportation sectors. Sean Brady is president and Andrew Tait, chief technology officer. Typical engagements involve strategic planning, app development, multi-criteria decision support, value analysis, benchmarking and target setting, implementation planning, and conflict resolution. Prism maximizes the human potential of client groups to make and implement decisions that achieve their preferred future.

Media Contact: Sean Brady
info@prismdecision.com
607.771.5411

Pint-sized but leonine, she was playing me as her favorite tune

Write down your plan and read it at least once a day

But first, let’s start with an anecdote.

As a young father, I found myself caught in a power struggle with my pint-sized but leonine daughter. And – no surprise to any parent – I was losing. Terribly. She had me completely figured and was playing me as her favorite tune. She’d call – seeking my attention by displaying negative behavior – and I’d respond – with angry discipline. The pattern was strong, established and escalating. I knew I was in serious trouble when I found her leaning against the screen in a second-floor window.

Coincidentally, I was soon off to the Creative Problem Solving Institute. I decided that I needed to make significant change and transform my relationship with my daughter. Good fortune had it that I went to a session by Roger Firestien. As I recall, he was presenting how to apply the creative problem-solving process (CPS) to personal transformation. He reviewed the standard CPS steps, with which I was intimately familiar:

• Identify a goal, wish or problem
• Gather data
• Clarify the problem
• Generate solutions
• Select solutions
• Plan and act

And then he recommended a strategy I hadn’t heard before: if you really want to implement your plan, write it down and read it at least once a day for two weeks. He guaranteed that fidelity to that final piece of advice would result in success in much less time than that.

Upon my return home, I did just that. I wrote the plan down and then I read it every morning when I got to work, before lunch and before I left for home. Although I read the plan for two weeks straight as recommended, the desired change occurred after about four or five days. I had transformed my relationship with my daughter.

Tierney’s strategies to maintain will power

Tierney’s strategies are based on Roy F. Baumeister’s recent research on will power, which “social scientists no longer regard as simply a metaphor. They’ve recently reported that willpower is a real form of mental energy, powered by glucose in the bloodstream, which is used up as you exert self-control…He and many of his colleagues have concluded that the way to keep a New Year’s resolution is to anticipate the limits of your willpower.”

Here are Tierney’s strategies for managing your limited will power and avoiding “ego depletion” and failure to change:

Set a single, clear goal: Set “a specific goal…and limit yourself to one big resolution at a time…With a finite supply of willpower, it’s tough enough to keep one resolution.”

Precommit: Have a plan and “further bind yourself by e-mailing your goal to friends or posting it on Facebook.”

Father and daughter 20 years on - Close as can be

Outsource: Use newly available tools such as Twitter or websites such as stickk.com. There you can make a formal contract,  identify a “referee” and even put money on the line. “The more you precommit, the better you do, according to stickK’s analysis of 125,000 contracts over the past three years. The success rate for people who don’t name a referee or set financial stakes is only 29 percent, but it rises to 59 percent when there’s a referee and to 71.5 percent when there’s money at stake. And when a contract includes a referee and financial stakes, the success rate is nearly 80 percent.”

Keep track: Monitor your progress continually.  Again, use the emerging tool sets available to you: “Entrepreneurs are rushing to monitor just about every aspect of your life — your health, your moods, your sleep — and you can find dozens of their products by consulting Web sites like Quantified Self and Lifehacker.”

Don’t overreact to a lapse: Avoid the “phenomenon formally known as ‘counterregulatory eating’ — and informally as the ‘what the hell effect.’” That is, don’t use a lapse as an excuse to give in completely and abandon your plan. You know, like the dieter who succumbs, eats a bowl of ice cream and then figures, “Hell, I may as well eat the whole carton.”

Reward often:  Finally, reward yourself often. As Tierney says, “If you use willpower only to deny yourself pleasures, it becomes a grim, thankless form of defense. But when you use it to gain something, you can wring pleasure out of the dreariest tasks.”

Reversal

The parents among you may be curious to know how I transformed my relationship with my daughter. I used a basic creativity technique called reversal. Each time the call came, my response was the exact opposite to what it had been. Instead of angry disciple, I gave my daughter a warm hug and told her I loved her. Simple as that.

CohortTracker™ is an early warning web application that allows schools to monitor students’ progress towards graduation. Educators can identify students getting off track and intervene before it is too late.

The app is designed to be a general-purpose student achievement tracking system that can be tailored to meet the specific needs of any education department. Simple, intuitive and easy to use, CohortTracker™ is now available. In fact, at a recent user feedback session, a delighted Deb Switzer (Odessa-Montour CSD) exclaimed that CohortTracker™ “is the easiest software I have ever used.”

CohortTracker™ is a small app requiring small training that elegantly delivers big data to support big decisions. If you would like CohortTracker™ to be available in your state, province or region, contact Sean Brady (info@prismdecision.com).

For more information see these additional posts:

• CohortTracker™ press release
• CohortTracker™ early warning system

In the late '90s, I had a colleague who was fond of saying, “When an operations analyst encounters a problem, he opens a spreadsheet. Now he has two problems…”

Indeed.

If you are responsible for a project that has a spreadsheet as a deliverable, go and suspend it now. Yes, now. Go on. When you return I'll explain why you had to do it.

Done? OK, I'll continue.

First, let me change tack for a moment. I'm actually a big fan of spreadsheets. I use them regularly. Excel is a great application. However…

spreadsheets should not be used for "line of business" applications.

In fact, I'd go further:

spreadsheets should only be used by the person who created them.

And, sometimes, not even then. We've all seen spreadsheets like the one below.

This isn't a spreadsheet. It's a database. And there are applications designed specifically to handle this sort of data. They're called "databases".

The "spreadsheet as database" is a pet peeve of mine as I receive data in this format all the time—and it's always a major pain. Consistency is hard to maintain when storing data in spreadsheets. You end up with customer IDs in the "Orders" sheet that can't be found in the "Customers” sheet. Or a sales tax of "Smith". A particularly pernicious problem with using spreadsheets in this manner is their tendency to autoformat fields. If I had a dime for every time a numeric customer ID with leading zeros had been formatted as 6.72512E+11…or an order had been fulfilled during the reign of Queen Victoria…

Also, there's a significant risk of messing up your data when it's held in a spreadsheet. Miss the fact that a sort hasn't included all of your columns and your records could be rendered worthless.

Even if you're maintaining a small database for your own use there are more appropriate solutions than using a spreadsheet. You don't have to become an Oracle database administrator. Personal database products like Bento are as easy to use as a spreadsheet. If you have a spreadsheet "database" in a corporate setting, and it's doing more than managing your lottery syndicate, then someone needs to be fired.

So, OK, it's not a good idea to use a spreadsheet in place of a database. But, spreadsheets are perfect for managing your accounts, right? Columns of numbers, balances, etc. Ideal.

Well, not really. The problem is that there are all sorts of rules and structure that can be used to help you manage your accounting—and spreadsheets don't utilize any of them out of the box. A specialized accounting application will check that accounts balance, report outstanding invoices/bills, calculate taxes, etc. You can build some of these features into a spreadsheet, but why do that when the work has already been done by experts? Also, no-one is going to be au fait with your custom spreadsheet, so when it comes time to share the accounts with a colleague, or your accountant, there's scope for confusion and mistakes. If you use a professional accounting package, however, it will have a structure that is familiar to anyone who has experience of business accounting.

Another problem with the use of spreadsheets in accounting is accuracy—a problem that will pop up again and again in this article. It is all too easy for a newly added row not to be included in a summation formula. Or for a tax rate to be changed inconsistently.

Unless your spreadsheet is being used by your lemonade stand business to see if you can afford that hamster, buy Quickbooks.

This seems like an appropriate point to address, in more detail, the lack of accuracy in spreadsheets. Sarbanes-Oxley and similar laws have resulted in more scrutiny of financial planning systems—and consequently, more scrutiny of corporate spreadsheets. Research, such as that reported by Raymond Panko in "What We Know About Spreadsheet Errors", has found that most of the spreadsheets used by organizations contain errors—and that a considerable number of those errors are serious. In one case reported in Panko's research, the error would have caused a discrepancy of more than a billion dollars! Similarly, an article by Mark Ward in New Scientist reported that Coopers & Lybrand found errors in 90% of spreadsheets they audited. Finally, the European Spreadsheet Risks Interest Group maintains a list of horror stories about real-world losses due to spreadsheet errors. Don't read it with the lights off.

Why are there so many errors in spreadsheets? To be fair, spreadsheets aren't the only models that contain errors. We all know that software has its fair share of bugs. But the sheer number of spreadsheets, coupled with their "homespun" development, and the difficult of reviewing their logic, makes spreadsheet development the Wild West of the modeling community.

It's extremely difficult to develop accurate spreadsheet models. Professional software developers liken it to programming in ancient languages that rely on the notorious "goto" statement. And then there's the challenge of reviewing formulae that contain "variables" such as "JG271". Paradoxically, familiarity with spreadsheets encourages inexperienced developers to "have a go". While I applaud this adventurous spirit, it doesn't result in well-crafted models.

The software development community has invested extensively in languages, frameworks and tools to improve the quality of software. In comparison, relatively little work has been done to improve the quality of spreadsheet modeling—and the work that has been done has had minimal impact on spreadsheet users. People find it difficult to maintain an understanding of complex logic, so they need to break large models down into simpler components. This isn't as easy to do in a spreadsheet as in, say, a modern object-oriented programming language.

Spreadsheets become increasing complex when models contain conditional (if X then Y else Z) and looping (do X Y times) logic. All business analysts will have encountered Monte Carlo spreadsheets that had thousands of rows of automatically generated results—and involved lookup logic for using those results. Macros can be written to address some of the challenges of using spreadsheets, but extensive use of macros is a clear indication that a spreadsheet may not be the best platform for your modeling. You end up dealing with all the complexity of software development, but suffering the weaknesses of spreadsheets for your efforts.

It's also all too easy to enter bad data into spreadsheets. A wrong keystroke and a formula is replaced with a static value, rendering the calculations meaningless. Freeform data entry allows you to destroy a model in an instant. Protected cells can only do so much. And, once again, if you protect entire sheets and have users interact with the data exclusively through macros, you're not exploiting the native strengths of the spreadsheet platform in the first place.

Another (ab)use of spreadsheets that I'm compelled to address is their use in business models—strategic planning models, forecasts, simulations, what-if analyses, etc. These models tend to have an extended lifespan, over which they are "tweaked" extensively—by different analysts—and used as the basis for significant corporate decisions. These are probably the most challenge models of all to develop—and, therefore, the most prone to errors.

Consider this: how many business problems are actually a natural fit for a two- or three-dimensional grid of cells? The gap between the real-world and the modeling environment is known as the "impedance mismatch"—and, in spreadsheet modeling, it's generally huge. The gap is generally smaller when a modern programming language is employed, as the software designer can map objects to real-world entities. Higher impedance mismatches place higher demands on the analyst, in terms of understanding the model. So, when dealing with an unfamiliar model—either due to its being a while since the model was used, or because it was originally developed by someone else—the chances of mistakes being made are correlated with the impedance mismatch.

If you are building a business model, use a specialist tool (e.g. a simulation application) or a general purpose programming language. That "little spreadsheet model someone threw together in a few days" will eventually cost you dearly.

Hold on? If spreadsheets are so bad at everything, should we be using them at all?!

Who said they were bad at everything? Certainly not me. I'd even go so far as to paraphrase the NRA and say that spreadsheets don't create bad models—people create bad models. And spreadsheets are actually excellent for one very important business activity—prototyping. Prototyping is when you build a model for the purposes of exploring an idea. Or for checking a theory. It basically building a model that you'll discard once you have your answer. Prototypes have a short lifespan and are only used by (or in collaboration with) the original author. They are never used in production.

Spreadsheets excel (no pun intended) in this activity. They are very inclusive, as everyone knows how to use them. And because a lot of thinking about the process they are being used to study is going on, mistakes are less likely to have grave consequences. They can also form the design specification for a more formal model, as the subsequent development process should place more rigorous checks on the logic.

In fact, one wag suggested that spreadsheets would be used more effectively if the save function were removed. I have some sympathy…

OK. I'm guilty of having been a little provocative in this article. But, only a little. If you are using spreadsheets for anything more than individual prototyping in your organization I urge you to seriously consider replacing them with more suitable models.

Porting your spreadsheets to a modern programming language, for example, will:

• result in a design that is a more natural fit to the real-world problem, and, therefore, produce a more accurate model
• allow the use of standard testing tools to improve the quality of the implementation
• allow the use of source code control systems to manage on-going development

There's a stack of evidence that suggests your spreadsheet models are ticking time-bombs.

Please, don't "solve" one problem by making it into two.