Making a crowdsourced map

I recently got two requests within a week to set up a system for a crowdsourced map (a map where anyone can add content to the map, or at least specific people you invite). Since it’s relatively easy to do using free technology, I thought it might be helpful to create a guide explaining the process and providing some examples. This guide is quite long as I wanted it to provide enough detail for a relatively non-technical person to be able to complete the process.

Here’s how the map works once it is all set up:

1. Users fill out a form you create that specifies a location along with whatever other data you want to show on the map (e.g. http://usvegweek.com/in-your-area/add-event/).
2. If you like, you can require each submission to be manually approved before being added (to prevent inappropriate submissions).
3. The map will automatically update showing the new entries (e.g. http://fish.freeshell.org/vegmap/vegweek_backup.html) It can take up to a day for new entries to show up after being approved.
4. Optionally you can set up a web page that lists all of the data in another form (a table, or blocks of text e.g. http://fish.freeshell.org/vegmap/restaurantlist.html).

 This post is essentially an expansion on some excellent instructions provided by Google, so thanks to them for providing those. Here’s how to set up your system.

1.    Plan your form / process

Before you begin creating anything, it’s important to do some planning. Make sure everyone working on the project understands what this will and will not accomplish. Begin planning out your form(s), including what your audience is, whether you need multiple forms for different groups, and what data you need to collect. This doesn’t need to be totally finalized, because once you make the draft form it’s easy to tweak. If you plan to have two forms (with both sets of data displayed on the same map) plan them both now.

2.    Create a new gmail account

This process depends upon a gmail account, and since the way it syncs isn’t totally secure, you don’t want to use your real account. Create a new one by going to gmail.com and hitting create an account. Use a name that refers to your project and is easy to remember (e.g. usvegweek@gmail.com). Make sure to write down the password, and set the birthday / mobile phone / etc. for someone planning to stay involved (also write down that information. After you fill out the form, click Drive in the bar at the top of the screen (or click on https://drive.google.com)

3.    Create your form

Click the red CREATE button at left, and select Form. Hit “Get started” if prompted. Enter a title and pick a theme. This takes you to the form in edit mode. For more information on editing your form, see Google’s help page. I recommend using separate fields for street address, city, and state (in that order). Zip code is not needed for the map. This allows you to require each field (with a single field users may skip part of the address) and allows you to sort the data later on by city or state if desired. When done, click “Choose Response Destination” (below the insert menu), select “new spreadsheet” (the default name is fine) and hit create. Wait for the “setting up spreadsheet” text to change to “View responses” and click it. This will open a tab with the spreadsheet that will hold the responses from your form. Next, click File  | Share and enter the Google accounts of any people you want to be able to edit the form (for example, your normal account and the accounts of others who will be working on the form).

4.    Review and finalize your form

Before you proceed, it is essential that you finalize your form. A limitation of this process is that once you proceed beyond this step you will no longer be able to make changes to your form. As such, take some time to review the form with whoever needs to approve it, and make any changes they request (it may be easier to give them edit access so that they can make changes directly). Be clear that once you move on you will not be able to make any changes to the form without starting over from scratch. You may also wish to test the form out with a couple of sample users to make sure they understand the form and your instructions.

5.    Create your fusion table

Go back to your Google Drive page, hit the red “CREATE” button, and choose “Fusion Table.” If you have never used fusion tables before, when you hit CREATE choose “Connect More Apps” and choose “By Google” in the dropdown at upper left. Click on “Fusion Tables” and hit the blue “CONNECT+” button. Now when you hit the red CREATE button you can select “Fusion Table.”

In the dialog that pops up, choose “Google Spreadsheets” at left, put a check box next to your new spreadsheet with your form responses, and hit “select.” If it says “import failed” hit cancel and try again. If prompted, hit the “Grant Access” button. When the “import new table” dialog pops up, hit “next” and then “finish.” From the Edit menu, choose “Add Column”, and give it the name “location” (without quotes) and set the type to location as well (leave the format field alone).

Hit Save. Go to File | About this table, and copy the value next to “Encrypted ID:” (you’ll need this on the next step).

Hit the blue Share button, and add the Google accounts of any people who will need to be able to approve submissions (give them edit access). Optional: If you wish to be able to show data from your table in list form (here's an example), under Share hit Change next to Private, and select “Anyone with the link” instead.

6.    Set up your form to synchronize with the fusion table

NOTE: These instructions assume that you created separate fields for street address, city, and state as recommended above. If you used a single field for complete address instead, follow Google’s instructions for this section.

Go back to the spreadsheet that holds your form’s responses. From the Tools menu, select Script Editor. Hit close on the window that pops up. Go to File | Project Properties, and when prompted to rename project, enter a simple name like “map sync” and hit OK. Click the “project properties” button then “add row.”

First create a field called “username” (without quotes) and set the value to your dummy email address. Hit “add row” again, and this time the field is “password” and the value is your dummy email account’s password. Hit “add row” again, create a “docid” field where the value is the “encrypted ID” value from step 4 (above). Hit “add row” two more times. The fourth property is called “streetAddress” and the value is the name of the street address field from your form (e.g. “street address”). The fifth property is called “cityColumn” and the value is the name of your city field (e.g. “city” or “event city”). The sixth property is called “stateColumn” and the value is the name of your state field (e.g. “state” or “event state”). The seventh and final property is called “latlngColumn” and the value is “location”. Hit save.

Go to http://fish.freeshell.org/vegmap/google_FT_sync.js and copy everything. Go back to the script editor, select all of the placeholder lines of code and replace them with the code you copied. Hit the save icon. Choose “select function” from the dropdown in the gray bar and choose “onFormSubmit”. Click the small gray triangle in the gray bar (the “run” button). Click Authorize, then Close. Click the clock icon in the gray bar, and click the text that says “No triggers set up. Click here to add one now.” Leave the first box set to “onFormSubmit” and leave the second box set to “From spreadsheet” but change the third box to “On form submit”. Click save, save the code again, and close the tab with the code in it.

7.    Test the sync process

Go to your form, and select View | Live Form. Enter sample results and hit submit. Go to your fusion table and refresh the page, and you should see that the data from the form has appeared, and that there are coordinates in the location field. I recommend submitting at least 2-3 test submissions to have some sample data to play with in your map.

8.    Set up moderation for your form (optional)

This step will allow you to moderate submissions before they appear on the map (to prevent spam or any submission that you don’t want to show up). If you omit this step, submissions will show up on the map automatically (although you could always delete them later).

From your fusion table, select Edit | Add Column. Call the new field “Approved” and make it a text field. Click on the “Cards 1” tab. Click the arrow to the right of “Cards 1” and rename it “Pending Approval”. Click the blue Filter button and select the Approved column. Click the check box next to “no value”. This means that the data on this tab will show submissions that have not yet been approved, in a format that makes it easy to see all of the data before approval. Copy the URL from your browser, which will jump directly to this approval tab. Confirm that you have given edit permissions on the form to whoever needs to be able to approve submissions, and send them this link. Let your moderators know that to approve a submission, they need to click on the record they want to approve, then click the pencil icon and put the word “Yes” (without quotes, with a capital Y) in the Approved field. Also let them know that it takes about a day after they approve a submission before it actually shows up on the map. Note that you could use different text to approve each record (e.g. “y” or “yes”) but it would require modifying the map code later on.

I recommend approving one of your test records so that your map will not be blank when you set it up (as it will only show approved records).

9.    Set up the “info window” (popup) for the map

For this step, you will configure what users will see when they click on one of the points on the map. It will help to have a basic understanding of HTML, but for simple popups you can get by without by simply modifying the default HTML Google provides.

From your fusion table, click the Help menu, then “Back to classic look”. Then the Visualize menu and Map. Click a point to see what the default popup looks like. Click the “Configure info window” link right above the map. Click the custom tab. You will see that the default format is the name of each field in bold, then the content from that field, then a line break. You can make whatever modifications to this popup (in HTML format) that you wish. If you want to see an example,  here is the popup code behind the dining icons in http://fish.freeshell.org/vegmap/vegweek.html. Note that the popup I have set up allow the user to click on the address field to get directions via Google Maps. Once you have made the changes you want, hit save and click a point to see how it looks.

10.    Choose the icon for the map (optional)

If you don’t wish each submission to show up as a small red dot, follow this step. From your Fusion table’s classic view (you should still be there at the end of step 9) click “Configure styles”. Choose a new symbol if you wish and hit save. Otherwise, if none of the limited choices here are acceptable just hit cancel. In the upper-right corner of the screen hit “Switch to new look”.

If you wish to see more icon choices, go to https://www.google.com/fusiontables/DataSource?dsrcid=308519#map:id=3 and click on the marker you like and copy the marker name (e.g. “dining” for the white knife and fork icon). If you wish to use one of these you will set that up later.

11.    Set up email notification when users submit data (optional)

If you are moderating submissions, it will be helpful for you to know when a user has submitted new data so that you can review and approve it. From your form, go to Tools | Notification rules. Check the box next to “A user submits a form” and choose whether you want to get an email every time there’s a submission, or just a daily digest. This will send emails to your dummy account, so the next step is to get those emails forwarded to whoever will be moderating the submissions.

Note that if you’re short on time, instead of proceeding with these instructions you can simply instruct each moderator to go to the form (when logged in with their own account) and follow the instructions in the paragraph above. This lets you set up the forwarding for everyone at once. Go to gmail (you should still be logged in with your dummy account). Click the gear icon in the upper right and select Settings. Click on “Forwarding and POP/IMAP” and click “Add a forwarding address”. Enter the first moderator’s email (where they want to be notified that a form has been submitted), hit Next and then Proceed. If you have multiple moderators who wish to be notified, enter each email as a new forwarding address. Before you proceed each user will need to click a link in the email that Google will send them to confirm the forwarding.

If you only have a single moderator, on the Forwarding and POP/IMAP tab, just hit the radio button next to “Forward a copy of incoming mail to” and select the address (and you’re done with this step). Otherwise, if you have multiple moderators, make sure “Disable forwarding” is selected and click on the “Filters” tab.

Hit “Create a new filter”. Enter the dummy email address in the “To” field and hit “Create filter with this search”. Check the box next to “Forward it to” and select the first email address you wish to forward notifications to. Hit Create filter. Repeat the instructions in this paragraph for each moderator who wishes to be notified of new submissions.

12.    Set up the map

This step involves editing html and javascript. It does not necessarily require an understanding of how they work unless you wish to make customizations (which should be fairly easy).

Begin by choosing a map template to use from the examples I provide here. If none of these examples is exactly what you want you will need to customize your map using one of these as a starting point. If you plan to have two forms (with data from both on the map) select one of the templates for two forms below (even if you only have one form set up so far):

• Simplest map: just a map of the data submitted via your form without moderation.
• Simple map w/ moderation: same as above, but requiring you to approve submissions before they show up.
• Map of two forms: a simple map that combines data from two forms (both moderated). For example, I had separate forms for restaurants and individual events.
• Map of two forms w/ search: this map combines data from two forms (both moderated), and lets you search for data from either source by a single field (in this case, the name field).
• Map of two forms w/ zoom buttons: this map combines data from two forms (both moderated), and has buttons to jump to predefined regions on the map. This is the map I actually used when I went through this process myself; the buttons jump the user to the two areas where we expected to get the most submissions.
• Map of three forms w/ zoom buttons that filters old past events: this map is the same as the one above, but it adds another layer and also filters out events with a date in the past. This ensures that data is removed from the map as it becomes irrelevant, but note that putting in a range of dates in the date field prevents that event from being properly filtered out.

Click on the template you wish and download the source code (File menu then | Save as, or “save as page” depending on browser) to your computer. To make edits, I highly recommend a text editor like Notepad++ which makes it easy to see if you make a mistake with the formatting. Open the html file, and look for the words “EDIT THIS” in the comments to ensure that you modify each of the sections that require you to make changes. If you will display two sets of data on the map, start by just updating one of them for now.

When you’re done making changes, upload your html file to wherever you plan to host it. If you do not have a web server you will need to find a place to host these files. The easiest solution is probably publishing via the public folder in Dropbox. I don’t have instructions for setting this up, but the Dropbox instructions are clear and simple.

Once you have uploaded your html file, you should see a point or two on the map (assuming that you have approved some of your test data). If you do not see any data, make sure there is at least one approved record, wait 24 hours and try again (it takes up to a day for submissions to show up on the map once they are approved).

13.    Displaying your data in table or list format (optional)

Maps are great when you want to explore data by location, but sometimes it’s better to be able to see data in a list or table. As easy solution is to send people a link to your fusion table’s table view or a card view.

If you want to display the data one record at a time feel free to copy the source code from http://fish.freeshell.org/vegmap/restaurantlist.html. Note that this is pretty crude code, it sorts by state but that's it. You will need to make modifications for it to work with your data; as this requires a Google Maps API key and understanding of javascript, I’m not providing instructions here.

14.    Adding a second layer to the map from a second form (optional)

When I went through this process, the people I built the map for wanted two different forms with two different sets of data. However, they wanted both sets of data to be displayed on the same map. To set this up, simply go through steps 3-11 of the entire process above again (using the same dummy Google account). You should have already selected one of the templates that supports two forms, so go back to your html file and add your second layer to the map.

15.    Test the final map

Make sure that the map looks how you want it to, then send the link to whoever else needs to approve it. At this point, encourage others to test submitting data via the form, approving it, and make sure that the next day it is visible on the map.

16.    Write up the final instructions and publicize your forms

I recommend sending an email to yourself and any moderators with the links to the live forms, the forms in edit view, the links to approve submissions, and the link to the final map. The various components of this system can be confusing, so it will help to have them all in one place as a reference. You can also "star" each form and table in Google Drive to help.

Next, share the links to the live forms with your users. You can embed the forms in your own web page, or just send out the plain links Google provides. Congratulations! To see the whole thing in action, check out http://usvegweek.com/in-your-area/.