Wnbp’s Local Pulse We’re excited! To learn more about the Artisans’ Revival, tune in to Local Pulse this Saturday at 9:00 am when we’ll be talking with John Brown about it. And be sure to stop by on Saturday, August 6, when we’ll be broadcasting live on WNBP 1450AM & 106.1FM “The Legends!” from Inn Street.
Please scroll down to show #147 for interview with John Brown on the Inn Street Artisans Revival.
Critical Mass – Newburyport Commons; Are you excited yet? Hell ya!
Right now, we stand at a precipice of critical mass in creating history.
There are two things that I need to please ask each and everyone of you to do right now, or as soon as you can.
So Im going to get right to the point.
There is now an Inn Street Artisans Revival Facebook page.
I need everyone to please go there and like it. then share it.
Then please thank Liz U for do a fantastic job for stepping up and creating it and please support her in her ongoing efforts to promote this event all week long.
If you are not register, please do now.
A poll and conversation asking if you are excited about the Inn Street Artisans Revival has now been created by Joe Moakley Jr.
I need to please ask each and everyone of you to go to this poll and tell Joe, and the city, how excited, and proud, you are to be part of the first ever Yankee Homecoming Inn Street Artisans Revival 2016, NO MATTER THE OUTCOME.
I said it before, and Ill say it again. History is in the making.
And you are a part of it. Its something to be proud of, and don’t ever forget it!
BRYAN EATON/Staff photo John William Brown is on a committee that worked to bring more local vendors and local artists to locations on Inn Street during Newburyport’s Yankee Homecoming instead of out-of-town vendors and sales displays.
Today begins an intensive promotional campaign with a featured article in todays Newburyport Daily News. It will be followed up Saturday morning just prior to the opening commencement of the Revival in a live broadcast interview with your’s truly on the WNBP’s Local Pulse Radio Show with Joe Dibiase at 9.a.m..
Since there will be numerous changes throughout the week with all the different Artisans, Musicians, Non-Profits and Performers that are scheduled, there will be plenty opportunity by the local promoters and press to feature different stories of all the diversity that the Inn Street Artisans Revival plans to bring to the area and festival.
I will be making a special effort to make sure the names of every Performer, Artisan and Participant of the Revival is given his due credit in coming forward and claiming their seat in the First Annual Inn Street Artisans Revival. There will be a follow up news article during the festival next week, as well as a live broadcast by Joe Dibiase at the Inn Street Stage on Saturday Morning, August 6th.
I am doing everything I can to make sure the schedules for Sunday and beyond are posted as SOON AS POSSIBLE! Please bare with me.
This is only the beginning. For those that have never did Yankee Homecoming before and are in it for the long haul, remember this. From over ten years of experience doing this festival, my best suggestion is take it slow and pace yourself. Drink plenty of water and try to get the right amount of sleep! And lastly, but most importantly, try to have fun.
If there is anything you need, please let me know. I am here…for you.
Because I’m gruff, I’m tough, and I’m ready to do battle!
We are taking it to the Street!
Yankee Homecoming is known as a weeklong event that caters to thousands of visitors.
But local artisans will get their due when the 59th edition unfolds this weekend.
A program called Inn Street Artisans Revival has been developed to make sure local artists, craftspeople and musicians get good billing in a valued local spot — Inn Street.
Organizers say attention to locals was championed because last year numerous vendors from national food chains and/or sales teams had prominent locations on Inn Street or Market Square.
Some local vendors expressed concern, pointing out that Yankee Homecoming started as an event by local talent.
So this year, Inn Street will be populated with many booths run by those from the Newburyport area.
“We want local artisans to have a good chance at success,” said John William Brown, a photographer who has spearheaded a greater recognition of local talent on Inn Street.
Yankee Homecoming starts Saturday and runs through Aug. 7.
Inn Street is a coveted locale for small businesses, and Homecoming organizers are encouraging the Inn Street Artisans Revival so local participants can find an audience.
Presentations will include appearances by talented youth artists. The Artisans Revival will also host representatives of art, crafts and healing arts.
There will be an Inn Street Farmers Market and Culinary Arts.
Paul Swindlehurst, general chair of Yankee Homecoming, said, “Last year we did receive some criticism from local businesses about the Inn Street programming, particularly regarding out-of-town vendors. This year we headed in a whole new direction.
“John Brown stepped in and has an amazing group of local artisans, performers and small businesses that will hew closer to the original intent of Yankee Homecoming — a home-grown festival.
Also, local nonprofit and humanitarian organizations will be hosted.
The “Revival” starts Saturday about noon with a sacred circle dance led by Ellen Kennedy.
Other artists and/or performers Saturday are scheduled to include: ArtyParty, with Diane Soares; Endless Energy, Jeff Cohen; Plum Island art, Sandra Turner; Amy’s Creations, Amy Sciuto; At the Beach, Darlene Grady Duggan; Golden Closet, Laura and Megan Golden; Angela Chiklis; Jafra, Deb Pynknowski; Fun Face Express (face-painting by kids); Ariana Bahret, lifeline practitioner; and culinary arts, at the nearby Greek Church, John Tasakirgis.
Brown will be offering high-end photos at his stand near Market Square. Many are artistic shots of the Merrimack River and other naturalist selections.
Organizers say that healing arts will be represented.
Among those slated to participate include Roberta Hoseman (crystal/healing stones); Kristin Robinson, reflexology; Laurie Perkins, animal and medium readings; Ariana Bahret, energy healing; and Michelle Lomasney, massage.
Diane Gronbeck, co-owner of the Brass Lyon gift shop at the base of Inn Street, said, “Another Yankee Homecoming is just about here, and all of us will be ready for the another year.
– Artisans/Musicians; Platform for precedent setting plan for collective contributions to non-profits and humanitarian groups.
You are receiving this email because you are either participating in the Yankee Homecoming Inn Street Artisans Revival, or have expressed interest in it. If you feel you have received this notification by mistake, or no longer want to receive these updates, simply reply with ‘Remove” in the subject line.
So we have the Artisans. We have the musicians. And we will be talking and featuring about all of them in the days, months, and year ahead. But before we do, lets talk a little more about the Non-Profits and Humanitarians. What are they, who they are, and their role in the Artisans Revival.
Or shall I say, the role of the Artisan Revival with the Non-Profits and Humanitarians.
In the Newburyport Revival presentation given on June 8th, (https://spiritofnewburyport.wordpress.com/2016/06/29/artisan-revival-library-presentation/) there was what I referred to as an ‘aha moment’. A ‘critical mass’ moment in which a certain ‘understanding’ had been ‘realized’. Earlier in the evening a switch had been made in the order of presentation of speakers because the ‘non-profit’ representative, Beecher Grogan of Lucys Love Bus, had to leave early. So, in a last minute switch, I changed speaker positions with Ben Twombly’s historical perspective, and gave the first slot to Beecher. One of the first things that Beecher said was, “I’m still not sure exactly what I am doing here, or what Lucy’s Love Bus has to do with the Artisans Revival, but I’m just going to ‘go with it’ for now.
Beecher Grogan and John Brown share the ‘aha moment’ at the Newburyport Library Inn Street Artisans Revival presentation, June 8th, 2016.
The point that I was trying to make is after all the controversy, opinions, perceptions, and talk, at the end of the day, I didn’t want to lose focus on what the ‘real conversation’ should be about. I want to remember this point for myself, as well as anyone else for that matter. I’ve made the better part of my living off of Inn Street for the last ten years. The Yankee Homecoming Inn Street Artisans Revival, above and beyond anything else, has given me an opportunity to give back to the community in which I have depicted in my photographs and artwork for many years now, in a way that I never thought I could. It is for the artisans, the musicians, the dancers, the poets, everyone, but none more so than the program I have been developing for the non-profits and humanitarians, especially for the children. I know for some, with all my screaming and perceptions of the splash pad and kids, that this may come as a surprise or seem hypocritical. But its not. And if anyone has really listened to what I have had to say, you would clearly understand this. There is a place and time for everything and everyone. And with that said, it is now time to bring forth and clarify the Inn Street Artisans Revival’s program for the non-profits and humanitarian groups, especially those regarding children.
Above all the talk lately, there has been a constant overlaying viewpoint….how do we make it all work together? Somehow, the Revival takes it all, jumbles it all up together, and throws it all out there on the street. No one really knows how it will all work out, and I suppose we wont until next Saturday, or perhaps, long after Sunday, August 7th, when the dust finally settles.
But until then, I would like to take this opportunity to elaborate on the Non-Profit and Humanitarian program of the Revival. Its funny how things work out sometimes. Whats really interesting at this stage is that, the opening of the Revival is less than a few days away, and its still all being worked out. Cancellations, alterations, and other things have led to a ‘rethinking’ and utilization of the Inn Street area courtyard. The original plan for a farmers market and agricultural element of the Revival has all but been canceled. This is unfortunate because I had high hopes of this element and the true ‘Revival’ concept, to me, is not, nor will it ever be complete, without it. If the Revival takes root and continues to grow in its foundation for years to come, then the agriculture element must take root. (Pardon the pun). And if anyone is reading this and understands the plight and role of the agriculture and farming industry in this community, I hope that someone stands up, and comes forward in the years to come and make this happen. Maybe next year.
But the reason why I bring this up now is that it directly had to do with how the Non-Profit and Humanitarian platform has developed. Byron’s courtyard off of Inn Street had been ‘bartered for’ with the Market Square Craft Day organizer in order to raise more money and lend a better avenue for implementing the Farmers Market. Since the farmers market never got off the ground, I had to switch up some planning and began experimenting with the idea of using it as a ‘non-profit and humanitarian’ collective exhibit. Its funny sometimes how things come together. Sometimes the best plans are not planned at all, and evolve. Its a matter of going with it, and allowing things to develop on their own.
I even hesitate to be putting this all out there because it hasn’t fully come together yet, and everything depends on others to come forth and MAKE it happen. As a coordinator, you can only do so much. You can create the original idea, but you must allow it to grow and have the faith in others to stand up, come forward and FOLLOW through with what they say they will do. And until that happens, you can only plan so much.
Once again, its funny how things play themselves out. I have a friend who was one of the very first vendors to book a spot at the Revival. For months she struggled with her business and the commitment that would need to be taken to participate. She finally had to let it go just a couple weeks ago and back out. She wrote me and told me she couldn’t make it, but had someone that was willing to take her spot.
As I write this, I just got a phone call from Dyke Henderson from the Daily News. He’s doing a featured story on the Inn Street Artisans Revival. Set up an appointment for a photo opp at my kiosk on Inn Street for 3pm this afternoon. I told him it was so good to hear from him because personally, I believe this is the story of the year. He agreed. There will be more to this story and its glad to see the local press covering it.
Now, back to the story behind the story.
So Micheal contacts me and we instantly hit it off. Our visions were aligned and we talked about how we could collaborate on creating a fundraiser for the non-profits and humanitarian groups. He employs kids and trains them in face painting. He is going to be setting up his 10×20 tent in Byrons courtyard and we have decided to turn what was originally going to be an area for the farmers market into a ‘non-profit and humanitarian’ courtyard exhibit, anchored by Fun Face Express and kids painting kids faces supporting the non-profits and humanitarian groups participating in the Inn Street Artisans Revival by donating 10% of their proceeds.
Each non-profit that will be in the courtyard will be doing their own fundraising project. The featured non-profit on the opening day is Lucys Love Bus, coordinated by Kat Couree and her daughter Courtney. They will have their own face painter, Julie King of Samarra Painting, as well as conducting other creative projects along the lines of a children’s based creative art fundraising. They came forward and played an important role with these ideas and were instrumental in sewing the seeds for later development for the non-profit and humanitarian element of the Inn Street Artisans Revival. They have beautiful hearts and I highly encourage anyone to come on out to Byrons courtyard on Saturday and support this effort. Please talk about it at your booths, spread the message, along with your best sales pitches for the duration of the festival!
Because this was only the beginning.
After that first ‘aha moment’ at the library, things began to really snowball after that first meeting with Mike Trombly and Fun Face Express. There are going to be some great fundraisers being conducted in Byrons Courtyard all week long including Opportunity Works, Order of the Eastern Star, Big Ideas and Small Solutions, The Pink House Project, The Newburyport Artist Association, and the Greater Newburyport Village, just to name a few. Ill get into the some specifics of these organizations and projects in part two of this article.
But for now I just want to end this section by expanding the fundraising foundation awareness by including a voluntary only platform of Artisans, and Musicians by suggesting to conduct raffles, auctions, and the like in order to make one contribution to a non profit and humanitarian project group participating in the first ever Yankee Homecoming Inn Street Artisans Revival. This is not a requirement or expected of anyone. It is simply a volunteer effort, reflecting back the spirit of many volunteer efforts that come together to make Yankee Homecoming the festival that it is.
Ive learned so much this year about these people behind the scenes, and myself. The idea to send a big collective Artisan ‘win win’ message, that this is what the Revival is all about and this is the conversation that we should be having down on Inn Street.
Stay tuned for part two to an unprecedented Artisans Revival Voluntary Platform for Non-Profit and Humanitarian Fundraising activities.
Hope you are all getting excited for the start of the Inn Street Artisans Revival.
Things have really stacked up the past couple weeks and have not been able to do half as much work as I wanted to. But everything is being done in order to establish as much of a promotional foundation for the artisans, musicians, and non-profits as possible. A facebook page as been created and will be published any day. The foundation for the Yankee Homecoming web page has been established and I will make sure that every effort to post each and every participant before the start on Saturday. Every effort will be made to correct and misprints and links prior to the start. If your listing or ‘day in the sun’ is not created or posted before Saturday, every effort is being made to get it done prior to your scheduled day.
feature pages are now being worked on and will be posted as soon as possible and notifications will be sent once the publishing’s are made.
Schedules and postings will remain on the Yankee Homecoming page for the entire year, and will serve as the foundation to build off of in the future.
Just a couple quick notices for now. I will be sending out some extensive announcements shortly.
In the meantime, I just wanted to address a few guidelines beginning this weekend.
All spaces will be designated at the beginning of each day. I will be located at my Spirit of Newburyport (located at the rear entrance of Orange leaf on Inn Street) kiosk every morning at 7 a.m. Please report there to be assigned your location. If I am not there when you show up, I will be on Inn Street somewhere and will return there shortly. PLEASE have your booth set up by 10 a.m. every day. You must remain set up until at least 5 PM. Many artisans will be staying later and until I receive further notice, you may stay as late as you want. I myself will be staying till at least 9PM.
You may access Inn Street via the Inn Street parking lot off of Green Street. You may park curbside to unload. You may park in the paid parking lot, or use the waterfront parking lot off of Water Street.
Opening commencements will begin at noon on Saturday Morning. Again, I will be on Inn Street at 7 a.m., but have to leave to be at wnbp radio studios for an interview by 8:45. I will return at 9:15. There will be someone else present on Inn Street in the interim to assist with set ups ( I hope ; ). Bets bet, please arrive by 8 a.m.
I am making appointments all week with ANYONE that would like to go over their placements prior to set up. Don’t hesitate to call me.
Joe Dibiase of wnbp ‘Local Pulse’ Saturday morning show will also be doing a ‘live broadcast’ from the Inn Street Stage on Saturday Morning, August 6th. You heard it first right here.
There may be also some ‘firsts’ being announced over the next couple days.
I had some issues with my email over the past week. If you have tried to get in touch with me and I have not responded, please try again.
There are a few I have received and have not been able to return as of yet. Please be patient. I expected a last minute rush on things, but nothing ever prepared me for what has come out of this.
Im sure there are some guidelines that I have forgotten to include here so I am sure I will be posting another.
I also have two MAJOR publishing’s to post very shortly. One having to do with the format of an unprecedented opportunity platform being created by and for the Artisans and Musicians to lead a foundation of fundraising for the non profits and humanitarian groups of the Inn Street Artisans Revival. The outline for Byron’s Courtyard to be used for a non-profit and humanitarian fundraising ‘center’ has been set in place. Please stay tuned for this announcement if you wish to be part of this unprecedented move for an opportunity to set standards for an ‘Artisan Led’ fundraising platform for years to come.
I am also hoping to make one more MAJOR announcement prior to the kick off on Saturday. But keeping my fingers crossed that it pans out. One way or another, I will be talking about the ‘behind the scene’ plans that have been in the works in order to bring a major fundraising effort to the Newburyport waterfront, unlike anything ever seen here before. So if it doesn’t happen this year, we will try again next.
Either way, it has been an amazing effort by many sources to bring this all together. It has been exciting to say the least and I am both humbled and honored to have been a part of it all, regardless of the outcome.
More to come shortly, please stay tuned and know that every possible is being done in order to deliver the best promotion, coverage, and implementation of the first ever Yankee Homecoming Inn Street Artisans Revival.
If anyone has any questions, concerns or suggestions at all, please don’t hesitate to call or write.
As a photographer and artist who has lived in Newburyport since 1991 and who has owned a gallery on Inn Street for 8 years, I must say that I am very disappointed to see such a large increase in vending license fee. This change alone could be the last nail in the coffin for artists who don’t have a brick & mortor location in this city to show their work. Since moving here, I have listen to artists tell stories of being pushed from location to location because of luxury condo development, and now the increase in the license fee will surely keep them from ever establishing a presence on Inn Street, or any other outdoor location in this city for that matter. John Brown is correct when he talks about economic development being based around an “art” economy. Take a look at just two communities, Rockport, Ma, and Ogunqit, ME. They both have a huge art presence which brings in thousands of people who eat in the restaurants, and spend $$$ in all the other local shops. Even though Newburyport recently became a “Cultural District”, I haven’t seen any other overt effort by the city to support and advertise it’s art economy, other than the occasional comment about art from local officials. It’s almost like the local art economy doesn’t exist as far as the city is concerned.
In the eight years I’ve had my gallery, not one local city official has come into my gallery to ask me how things are going, ask me what the city can do better to promote the art community, or to buy a piece of art or a photo. The Mayor was in eight years ago when I had a Chamber ribbon cutting, but not since. And with all due respect to the Mayor, in contrast, Mayor Gray (and his wife) from Amesbury attends as many art openings and events as he possibly can because he understands how important an art economy is to the local economy. I know this because I have cosponsored art events in Amesbury and have talked with Mayor Gray at every event and listened to him expound on how great the the events are and how important it is for a locality to support it’s artists. If you’re inclined, give him a call and tell him I encouraged you to call.
About the only time I see any visible presence by a city official in an official capacity at an art event din Newburyport, is at the NAA, which is understandable because of the importance of that organization. But there are other galleries in this city that those same officials never go to! I currently have an exhibition of Harold Feinstein’s work, a very important American photographer (www.sweethavengallery.com, www.haroldfeinstein.com) who passed away last year and was a resident of Merrimac. I advertised in the Daily News, The Current, The Boston Globe North, and on line. Although not huge in size, this exhibit is huge in scope with Harold’s most iconic images which include original signed prints. Many people from out of town have come to see his work because of the advertising, but not one city official or head of any other major organization in this city has stepped through the door. We had an event on the 29th at the Firehouse which also got all the coverage I mentioned above, about Harold’s life and work, and it was a great success! But sans any city official in an official capacity showing support for the arts.
Every major city in this country and many, many small communities as well, support the arts in their communities to the best of their ability because they get it! Sad to say, but I don’t think this city really gets it. If you think I’m wrong, I’d love to hear your perspective.
What I’ve said isn’t meant as a complaint. I’m saying all of this as a matter of how I have experienced the art scene in Newburyport, including speaking with many other artists and how they feel and what they have experienced. Speaking as an artist, art is what we live for, it is who we are, we cannot live without it, and we want to share it with the community we live in and with the world. But if the city continues to give the art economy short shrift, how is that economy going to survive? This is what it says on this page (http://www.cityofnewburyport.com/office-of-arts-tourism-and-cultural-affairs) of the city web site “Welcome to Newburyport’s dynamic community of artists, award-winning writers, performers and craftsmen and women. Our thirteen art galleries, four museums, four theaters, four cultural centers and twelve performance spaces help in the creation of an energetic, vital streetscape for visitors and residents.” That’s basically all that is said about the arts in Newburyport. On this page (http://www.cityofnewburyport.com/office-of-arts-tourism-and-cultural-affairs/pages/annual-cultural-events) it lists cultural events for the year, including an event at the NAA and the Literary Festival. Not one mention of the 4 ArtWalk weekends which 12 galleries and 8 local business partners that support the arts participate in. Really?
And now the city wants to more than double the fee for the transient vendors license which is all but going to eliminate any possibility for any artist to have a presence on Inn Street. You might as well build a wall around Inn Street with a sign that says “Artists not welcome.” As I said earlier, the last nail in the coffin. John Brown has a great idea for turning Inn Street into a thriving arts scene as well as to make it a visual beautiful place that people will want to visit, unlike what it looks like now. Please consider what he has to say and what his vision is. It makes sense.
I implore you to reconsider how the license fee is calculated and to consider a sliding scale fee that is more appropriate to the type of business applying for a license. This community needs more art, not less. More art will bring in more people who will spend more $$$ in other businesses. If the fee is set to $1500, you will never see any artists on Inn Street again. Please, give that some thought when making your final decision.
Bcc’d are all in attendance at the first License and Permits Committee meeting discussing the Transient Vendors Licenses.
Forwarded here is the email from Councilor Eigerman which recaps of the meeting held on 7/6. His notes reflect well the concerns, questions and forward approach desired by those in attendance. If there is more to be added, please send them along and I will see that they are distributed.
In addition to the amended and comparison versions of the ordinance, please find an email letter from Jon Brown and notes from Ginny Roberts on the previous version.
I will be out of town until Tuesday, July 19th and will look to set up another meeting before the end of the month.
Please let me know your thoughts, concerns or questions.
As a result of the meeting held by the Committee on Licenses and Permits on July 6, here is another draft of the transient vending ordinance, plus a redline against the June 23 version you all had in front of you that night.
“Peddlers” are already covered elsewhere in our municpal code.
e.g. ice cream trucks because they stop for only 15 minutes at a time, and usually less.
Food/beverages are treated same as merchandise for purpose of these licenses.
However, ordinance clarifies that board of health permits may also be required, as already specified in other parts of our municipal code.
Similarly, if another part of our municipal code requires, say, a parks commission permit, that is in addition to a transient vendor license.
Instead of up to three licenses on Inn Street, the Licensing Commission could grant up to five licenses citywide.
Up to three licenses outside of downtown.
Parked vehicles would not be allowed on Inn Street, so that means up to three parked vehicles citywide (but not downtown).
The normal course is for the City Clerk to accept applications for the coming April-December season during the month of February.
Copy of application goes to city boards/commissions whose property would be specially affected (e.g. food truck parked in front of Atkinson Common), and they must send comments to License Commission within 20 days.
The License Commission’s decision on the applications is due by March 31, which is enough time for them to get ready.
The City Clerk publishes, posts, and mails notice of the public hearing.
The application processing fee is only $50 + $1 per property owner within 300 feet (who must get mailed notice).
e.g. Greater Newburyport Chamber of Commerce, and the Yankee Homecoming Committee.
There is no $1,500 fee for such special licenses to charities.
The proposed amendments would add another $2 million of general comprehensive required for vehicles, due to the greater/different risks they pose.
Must staff their locations during licensed hours (max. remains 8:00 am to 9:00 pm), so location is not just advertising.
Must post a menu or price list.
Must either remove their vehicle/cart/structure at closing, or secure it in place, somehow.
Power remains with License Commission to revoke for specified violations.
More types of violations listed.
To afford post-deprivation due process, revokee may still appeal to the License Commission at a properly noticed, public hearing.
If a license is abandoned, surrendered, or revoked in season (April-December), then city clerk publishes, posts, and mails notice of its availability.
Clerk can take applications for all or part of the unexpired term, on a rolling basis.
License Commission must act on such rolling applications within 30 days of receipt, and need not wait for multiple applicants to file.
By writing a letter and paying $10 by the end of February, anyone can get notice of any and all transient vendor licenses that become available in a given year, including special licenses granted by the City Council, and also in season (April-December) abandonments, surrenders or revocations.
You are receiving this email due to your request or interest in the musical and street performer element of the Yankee Homecoming Inn Street Artisans Revival. If you have ended up on this list by mistake, or if you choose to no longer receive these updates, please reply with ‘REMOVE” in the subject line.
This update and notice has been initially sent to the ‘Musical and Street performer’ list of the Revival, but wanted to forward it to the Artisans as well in case any of you are interested or have friends that are willing to participate in this segment of the Inn Street Artisans Revival as well.
The Yankee Homecoming Inn Street Stage is now under the direction of the Inn Street Artisans Revival. For the past few months I have been collecting a talent pool of performers that have been interested in participating initially in the Revival segment beginning on Saturday, July 30th, 2016 thru August 7th. The Inn Street stage is being conducted and designed to be utilized as a ‘complimentary’ stage to the main Market Square stage. In other words, what we don’t want to create, as in years past, is a ‘conflict in sound’ between the two stages, especially due to their close proximity. There has also been concern among some brick and mortar businesses of the area in regards to ‘sound reverberation’. These conflicts have stymied the Inn Street Stage and its musical performances in the past few years. The effort of the Revival is to instead of phasing out and allowing the Inn Street musical portion of Yankee Homecoming to fizzle out and die altogether, is try to REVIVE IT, and give it some new life with a fresher and more logistical approach.
As someone who is exposed to the many great musical buskers of the year, I can attest to the many talented performers I see day after day who make their presence known by ‘singing and playing from the heart’. This is what I am after, and looking for when I am looking to book talent for the Inn Street stage. After the year of marketing and promoting for this segment of Yankee Homecoming, I am convinced that we can indeed ‘revive’ the Inn Street Stage and not only book it as a ‘complimentary’ performance stage, but to begin to turn the tide into a main source for some great alternative musical acts that are not available to be seen nor heard through the other musical stages of the Yankee Homecoming lineup.
1. limit the performance times to the ‘break down and set up’ times conducted by the main Market Square stage, (Half Hour performances) to begin with DURING THE MARKET SQUARE PERFORMANCES ONLY. Once the Market square performwnces are over for the day then we will have a different approach to scheduling.
3. have no more the two instruments amped at any one time, with NO BASS OR PERCUSSION’S. INITIALLY.
The scheduling for the Inn Street Sage will begin on Saturday 30th. We are looking to debut the Revival on that day even though most Yankee Homecoming events as well as musical performances are not scheduled to begin until Sunday, July 31st.
If you look at the Market Square Stage Schedule (link below), you will see the performance times. Performances are scheduled to run for one hour beginning at 11:00 A.M. till 5PM daily. At the end of the hour there will be a fifteen minute time allotment for breakdown of existing set, then a fifteen minute set up time allotment for the next performance. This half hour hour slot will be for scheduling Inn Street performances. However, you will note that not all time slots have been filled, and I would expect that not all slots will be filled by the start of the festival.
So this is how I would like to approach scheduling for the Inn Street Stage. I would like to begin with everyone forwarding as many days and time slots as they would prefer and capable of playing from July 30th to August 7th. I will then go from there to see how many of the slots I can fill. I would like to limit the slots initially to TWO a day. In other words, the initial limit (subject to change) for any one performer will be TWO, ONE HALF HOUR performances during the allotted time schedules daily. So, since the Market square performances begin at 11 am, I would like to begin the ‘morning’ portion of the performances on Inn Street at 10 AM. These would be very mellow, instrumental, piano, guitar or new age genre of music. This morning segment will be for a full hour. Then, seeing the first performance is scheduled at Market Sq at 11-12:00 noon. the second time slot would be from 12:00 noon to 12:30 PM. The next at 1:30-2:00 PM. See how that works according to the market square Stage schedule? Now, as far as the evening portion of the Inn Street Stage, I would like to see performances going all through the evening until 9 PM seeing that there will be less conflicts with both stages and brick and mortar businesses. So if you are available to play during the evening, the schedule is open at this particular time. Please keep in mind that this is a first time event and things will have to be adjusted and tweaked as it goes along. But it is also a great opportunity to get in on a first time event that may have a great opportunity of becoming an annual event with opportunity’s for paid performances beginning next year. If I remain as coordinator of the Revival for next year, you have my promise that this is going to be my main goal for the musical and street performing segment of the Inn Street Artisans Revival 2017.
One last point regarding the Inn Street Stage. An effort is being made to change up the feel, look and flow of the Inn Street Stage by creating an ‘Inn Street COURT and MUSICAL STAGE/TENT’. These alterations are just now being put forth as we enter the actual layout of all booked artisans and musical performances. The idea is to create a non-liner flow of traffic (Not just up and down Inn Street) but criss-crossing back and forth from the plaza and Byron’s courtyard. The first adjustment will be the location of the stage itself that may be set forward some from its original placement further back in the Plaza. This will give performers the ability to play directly to the passing crowd as the walk up and down Inn Street. If however you succeed in drawing a crowd that prefers to either sit at the fountain stairs or dance in the plaza, you will be able to directly face them there. Alongside the stage will be the ‘Musical Artisans Tent’ that hopefully will be manned by volunteers to peddle your ‘Music wares, c.d.’s, collect names and contacts for mailing list etc etc.”.
The other addition will be the placement of two food vending operations, the ‘Greek Church’ and Otto’s Pizza set back off the main Inn Street Pedestrian walkway (behind the Stage) and set in relative proximity to the stage in order to further create the concept of a ‘Courtyard’. This is designed to ‘draw’ people in off of Inn Street and invite them to ‘hang out’ and listen to the music. There are also plans to provide a limited number of ‘patio’ tables and chairs adjacent to the Stage area.
Across the stage area and adjacent to the food vendors, and alongside the back part of the fountain, will be a children’s workshop tent and table, an arty party.com (https://anartyparty.com/). There is also an invite out to WBNP radio to add to the courtyard placement. WNBP radio has been regular over the years on Inn Street doing live broadcast from their Inn Street Location. I am currently waiting to hear from the main director as to whether or not to accept this years invitation to participate in the first Yankee Homecoming Inn Street Artisans Revival. Regardless, plans for an alternative radio or internet broadcast is being made. The idea would be to have a source for live broadcast for music and interviews to be conducted from the Inn Street Stage Court.
So, in short, that’s it for this version of the first Inn Street Artisans Music and Performers Update for now. As you can see this is just the beginning and we are looking forward to creating an event that will ultimately be one of Yankee Homecomings highlights.
Thanks for your support and continued interest in the Revival. if you have any questions, concerns, suggestions, or ideas to contribute to this portion of the program Id be very interested in hearing about your ideas. Please check us out online and in the recently published full color Yankee Homecoming program book, as well as all the informative links below to gain a better understanding on the who, whats, wheres and when’s of the Yankee Homecoming Inn Street Artisans Revival!!
Please get back to me ASAP with your request for days and times of your performances. Thank you!
Also, please forward any and all information regarding your act so that I can utilize this information in all press release, web page and marketing outlets.
This is juts a short update regarding yesterday posting of the Revivals ‘Opening Day Line Up’. If I didn’t make this clear enough, I apologize. Yesterdays postings of musicians and artisans were ONLY for the Opening day commencement on July 30th. ALL artisans and participants will be listed, updated, posted and promoted in every way feasible and possible according to the dates you have been booked for.
Once again, I will remind anyone that I simply do not have te time to retrieve or go looking for your web sites, facebooks, etc. If you have not sent me ‘feature ‘ material, bios, or related material, Please do Now.
Musicians. For those that have not responded as of yet. Please get me your preferred dates and times asap. Than you.
I was sent the following this morning. I don’t know how finds them,…but she always has the most pertinent song to share with me.
Hang on. We are almost there.
Once again, for those that are new to this email list, if however you have ended up here by mistake or no longer wish to receive the Revival updates, please reply with ‘remove’ in the subject line.
It is my pleasure to announce the opening commencement for the Yankee Homecoming Inn Street Artisans Revival on July 30th, 2016 at 12:00 P.M. noon.
The Inn Street Artisans Revival is a first time event. It was aimed at empowering a more local based arts and crafts inn Street sponsorship of the largest festival in Newburyport, Yankee Homecoming. It is a collection of artisans, small business, healers, musicians, and non-profits and humanitarian groups from Newburyport and the surrounding area.
A separate page featuring the artisans in more detail will be presented shortly prior to the Opening of the Revival. There will be different Artisans featured on different days of the Revival, with upwards of over thirty artisans participating in the Revival overall. All scheduled appearances will be featured shortly.
For now I would like to give my thanks and appreciation for the initial core of the Artisans that have signed on for this first time event and for coming forward and claiming their space at the first Yankee Homecoming Inn Street Artisans Revival. Without you, this event would not have been possible.
More ‘Featured’ Artisans details and specifics to be posted shortly.
Pleas stay tuned for a complete listing shortly.
“We are revitalizing Yankee Homecoming 2016 with the original intent and vision of Jack Frost, the man and artist who envisioned the very first Yankee Homecoming in 1958. A festival that was motivated and engineered by local business, community, and volunteers to serve as a revival of local, arts and craft co-creative, community based-sustainable economics.
This dedication will mark the official opening of the Yankee Homecoming Inn Street Artisans Revival 2016.
Following the dedication will be a sacred circle dance led by Ellen Kennedy, representing the performing arts portion of the Revival.
At 1 P.M., featured musician, Lewis Flores, a local, yet, world renown violinist, will be kicking off the musical portion of the Artisans Revival. Lewis Flores has performed internationally with some of the foremost orchestras and philharmonics. Lewis has immense versatility as a performer across a broad spectrum of musical styles and is comfortable performing in venues from small intimate gatherings to concert halls. Lewis’ performances consistently earn him high praise for his ability to connect with the music and his audience.
I first saw Lewis from my Inn Street Art cart a couple years ago as he performed an impromptu street appearance at Market Square. I would have to say that the dream of seeing him perform as part of a ‘Buskers Stage’ on Inn Street began there. A couple weeks ago, my girlfriend (who practically hobbled across the street just to see him as she was preparing for back surgery) and I happened across a performance at Market Square once again. One of the traits that I wanted to see and hear from the musicians of the Revival was ‘playing from the heart’. I could not have seen someone who better personifies this trait. This time I invited him to play as part of the Yankee Homecoming Inn Street Artisans Revival. As it has turned out, he has now become the featured performer to kick off the Artisans Revival. If you enjoy music that personifies the ‘spirit of love and appreciation of the arts,..do not miss this performance. I will have more to feature regarding this performance, and others, in the days ahead. Please stay tuned.
A little more about Lewis, a local performer.
Lewis has been studying and playing the violin for more 20 years, having begun at the age of 9 with the Juan José Landaeta Youth Symphony Orchestra in Venezuela. Very early in his career, Lewis was asked to join the first violin section of the National Children’s Orchestra of Venezuela under the direction of Gustavo Medina and later Gregory Carreño. At the age of twelve, Lewis auditioned for the Antonio Laura Symphony Orchestra of Bolivar State and became one of the youngest concertmasters in the history of the orchestra. Lewis was blessed to participate in many national and international events during this time and toured with the orchestra in Columbia, Brazil, Costa Rica, London, and Italy.
In recent years, Lewis has performed as a soloist by invitation with many symphony orchestras and has been invited to participate in a number of masterclasses. He has worked with many prominent international maestros including Steven Mxioren (Vienna Philharmonic), Marilú Speaker (The New England Conservatory), Jose Francisco del Castillo (The Latin American Violin Academy) and Olaf Easling (Concertmaster of the Philharmonic Orchestra of Venezuela).
Currently, in addition to his work as a premier soloist, Lewis serves as the Artistic Conductor for the Juan José Landaeta Youth Chamber Orchestra and a Professor of Violin for both the Conservatory of Music Carlos Afandor Real and Bolivar City Youth Symphony Orchestra.
Also performing att he Inn Street Stage; The Flying Toaster Brothers 3 P.M.
Steve Nicolas (Salisbury MA) has been playing fingerstyle acoustic and electric guitar for nearly 40 years. He has recorded with the Vermont band The Ultra Brothers, and their album has had a lot of college radio play.
As a duo we are focused on instrumental duets from Duane Allman’s “Little Martha” to originals by Pete Huttlinger and adaptations of well known tunes such as “And I Love Her.” I accompany Steve on mandolin, fiddle, low D whistle and open tuned guitar. I also sing.
Patrick Hornig (Merrimac MA) I taught myself fiddle and won first place in the open division of the 19th annual Craftsbury VT Fiddle Contest after playing for only 6 months in 1981. I am a multi-instrumentalist and composer as well as producer. I was a two time finalist for Best Sound Design at the New Hampshire Theater Awards and produced a CD for the Shakespearean troupe Advice to the Players. The CD included my original works based on traditional English folk music and Early music as well as electronica .
Also for the past 30 plus years I have been a member of the band Speed the Plough and produced a self titled CD of traditional and original works in 2013.
There will be readings and sale of a featured book, and other arts and crafts for children.
at the Newburyport Library Presentation on June 8th, 2016.
Order of the Eastern Star is affiliated with the Masonic Lodge. They have active chapters in both Newburyport and Haverhill, MA and support youth groups for girls (International Order of the Rainbow for Girls) and boys (DeMolay). Their service to our communities includes collecting food and clothing for homeless shelters, raising money for the American Cancer Society, MDA, Diabetes Foundation, no-kill animal shelters and much more.
More will be featured on this special event, as well as all other artisans, Musicians, Non – Profits and Humanitarians and demonstration. Please stay tuned.


Let’s start with the disclosures: by most interpretations I work for a competitor to what Salesforce.com and VMWare are trying to do with VMforce. And all I know about VMforce is what I read in a few authoritative blogs by VMWare’s Steve Herrod, VMWare/SpringSource’s Rod Johnson and Salesforce’s Anshu Sharma. So no hard feeling if you jump off right now.
Overall, I like what I see. Let me put it this way. I am now a lot more likely to write an application on force.com than I was last week. How could this not be a good thing for SalesForce, me and others like me?
On the other hand, this is also not the major announcement that the “VMforce is coming” drum-roll had tried to make us expect. If you fell for it, then I guess you can be disappointed. I didn’t and I’m not (Phil Wainewright fell for it and yet isn’t disappointed, asserting that “VMforce.com redefines the PaaS landscape” for reasons not entirely clear to me even after reading his article).
The new thing is that force.com now supports an additional runtime, in addition to Apex. That new runtime uses the Java language, with the constraint that it is used via the Spring framework. Which is familiar territory to many developers. That’s it. That’s the VMforce announcement for all practical purposes from a user’s perspective. It’s a great step forward for force.com which was hampered by the non-standard nature of Apex, but it’s just a new runtime. All the other benefits that Anshu Sharma lists in his blog (search, reporting, mobile, integration, BPM, IdM, administration) are not new. They are the platform services that force.com offers to application writers, whether they use Apex or the new Java/Spring runtime.
It’s important to realize that there are two main parts to a full PaaS platform like force.com or Google App Engine. First there are application runtimes (Apex and now Java for force.com, Python and Java for GAE). They are language-dependent and you can have several of them to support different programming languages. Second are the platform services (reports, mobile, BPM, IdM etc for force.com as we saw above, mostly IdM for Google at this point) which are mostly language agnostic (beyond a library used to access them). I think of data storage (e.g. mySQL, force.com database, Google DataStore) as part of the runtime, but it’s on the edge of the grey zone. A third category is made of actual application services (e.g. the CRM web services out of SalesForce.com or the application services out of Google Apps) which I tend not to consider part of PaaS but again there are gray zones between application support services and application services. E.g. how domain-specific does your rule engine have to be before it moves from one category to the other?
As Umit Yalcinalp (who works for SalesForce) told me on Twitter “regardless of the runtime the devs using the Force.com db will get the same platform benefits, chatter, workflow, analytics”. What I called the platform services above. Which, really, is where most of the PaaS value lies anyway. A language runtime is just a starting point.
So where are VMWare and SpringSource in this picture? Well, from the point of view of the user nowhere, really. SalesForce could have built this platform themselves, using the Spring framework on top of Tomcat, WebLogic, JBoss… Itself running on any OS they want. With or without a hypervisor. These are all implementation details and are SalesForce’s problem, not ours as application developers.
It so happens that they have chosen to run this as a partnership with VMWare/SpringSource which makes a lot of sense from a portfolio/expertise perspective, of course. But this choice is not visible to the application developer making use of this platform. And it shouldn’t be. That’s the whole point of PaaS after all, that we don’t have to care.
VMWare wants us to know they are under the covers because of course they have much larger aspirations than to be a provider to SalesForce. They want to use this as a proof point to sell their SpringSource+VMWare stack in other settings, such as private clouds and other public cloud providers (modulo whatever exclusivity period may be in their contract with SalesForce). And VMforce, if it works well when it launches, is a great validation for this strategy. It’s natural that they want people to know that they are behind the curtain and can be called on to replicate this elsewhere.
But let’s be clear about what part they can replicate. It’s the Java/Spring language runtime and its underlying infrastructure. Not the platform services that are part of the SalesForce platform. Not an IdM solution, not a rules engine, not a business process engine, etc. We can expect that they are hard at work trying to fill these gaps, as the RabbitMQ acquisition illustrates, but for now all this comes from force.com and isn’t directly replicable. Which means that applications that use them aren’t quite so portable.
In his post, Steve Herrod quickly moves past the VMforce announcement to focus on the SpringSource+VMWare infrastructure part, the one he hopes to see multiplied everywhere. The key promise, from the developers’ perspective, is application portability. And while the use of Java+Spring definitely helps a lot in terms of code portability I see some promises in terms of data portability that will warrant scrutiny when VMforce actually rolls out: “you should be able to extract the code from the cloud it currently runs in and move it, along with its data, to another cloud choice”.
Does the code change depending on whether I am talking to a local relational DB in my private cloud or whether I am on VMforce and using the force.com database?
What about the other VMforce.com services (chatter, workflow, analytics…)? If I use them in my code can I keep using them once I migrate out of VMforce to a private cloud? Are they remotely invocable? Does the code change? And if I want to completely sever my links with SalesForce, can I find alternative implementations of these application platform services in my private cloud? Or from another public cloud provider? The answer to these is probably no, which means that you are only portable out of VMforce if you restrain yourself from using much of the value of the platform. It’s not even clear whether you can completely restrain yourself from using it, e.g. can you run on force.com without using their IdM system?
All these are hard questions. I am not blaming anyone for not answering them today since no-one does. But we shouldn’t sweep them under the rug. I am sure VMWare is working on finding workable compromises but I doubt it will be as simple, clean and portable as Steve Herrod implies. It’s funny how Steve and Anshu’s posts seem to reinforce and congratulate one another, until you realize that they are in large part talking about very different things. Anshu’s is almost entirely about the force.com application platform services (sprinkled with some weird Facebook envy), Steve’s is entirely about the application runtime and its infrastructure.
One thing that I am surprised not to see mentioned is the management aspect of the platform, especially considering the investment that SpringSource made in Hyperic. I can only assume that work is under way on this and that we’ll hear about it soon. One aspect of the management story that concerns me a bit is the lack of acknowledgment of the challenges of configuration management in a PaaS setting. Especially when I read Steve Herrod asserting that the VMWare/SpringSource PaaS platform is going to free us from the burden of “handling code modifications that may be required as the middleware versions change”. There seems to be a misconception that because the application administrators are not the ones doing the infrastructure updates they don’t need to worry about the impact of these updates on their application. Is Steve implying that the first release of the VMWare/SpringSource PaaS stack is going to be so perfect that the hypervisor, guest OS and app server will never have to be patched and versioned? If that’s not the case, then why are those patches suddenly less likely impact the application code? In fact the situation is even worse as the application administrator does not know which hypervisor/OS/middleware patches are being applied and when. They can’t test against the new version ahead of time for validation and they can’t make sure the change is scheduled during a non-critical period for their business. I wrote an entire blog post on this issue six months ago and it’s a little bit disheartening to see the issue flatly denied and ignored. Management is not just monitoring.
Here is another intriguing comment in Steve’s entry: “one of the key differentiators with EC2 based PaaS will be the efficiencies for the many-app model. Customers are frustrated with the need to buy a whole VM as the minimum service unit for their applications. Our PaaS will provide fine-grained resource separation”. I had to read it twice when I realized that the VMWare CTO was telling us that splitting a physical machine into VMs is not a good enough way to share its resources and that you really need middleware-level multi-tenancy. But who can disagree that a GAE-like architecture can support more low-traffic applications on the same server than anything based on VM-based sharing? Which (along with deep pockets) puts Google in position to offer free hosting for low-traffic applications, a great way to build adoption.
These are very early days in the history of PaaS. VMWare, like the rest of us, will need to tackle all these issues one by one. In the meantime, this is an interesting announcement and a noticeable milestone. Let’s just keep our eyes open on the incremental nature of progress and the long list of remaining issues.
There is the Cloud that provides value by requiring as few changes as possible. And there is the Cloud that provides value by raising the abstraction and operation level. The backward-compatible Cloud versus the forward-compatible Cloud.
The main selling point of the backward-compatible Cloud is that you can take your existing applications, tools, configurations, customizations, processes etc and transition them more or less as they are. It’s what allowed hypervisors to spread so quickly in the enterprise.
The main selling point of the forward-compatible Cloud is that you are more productive and focused. Fewer configuration items to worry about, fewer stack components to install/monitor/update, you can focus on your application and your business goals. You develop and manage at the level of application concepts, not systems. Bottom line, you write and deploy applications more quickly, cheaply and reliably.
To a large extent this maps to the distinction between IaaS and PaaS, but it’s not that simple. For example, a PaaS that endeavors to be a complete JEE environment is mainly aiming for the backward-compatible value proposition. On the other hand, EC2 spot instances, while part of the IaaS layer, are of the forward-compatible kind: not meant to run your current applications unchanged, but rather to give you ways to create applications that better align with your business goals.
Part of the confusion is that it’s sometimes unclear whether a given environment is aiming for forward-compatibility (and voluntary simplification) or whether its goal is backward-compatibility but it hasn’t yet achieved it. Take EC2 for example. At first it didn’t look much like a traditional datacenter, beyond the ability to create hosts. Then we got fixed IP, EBS, boot from EBS, etc and it got more and more realistic to run applications unchanged. But not quite, as this recent complaint by Hoff illustrates. He wants a lot more control on the network setup so he can deploy existing n-tier applications that have specific network topology/config requirements without re-engineering them. It’s a perfectly reasonable request, in the context of the backward-compatible Cloud value proposition. But one that will never be granted by a Cloud that aims for forward-compatibility.
Similarly, the forward-compatible Cloud doesn’t always successfully abstract away lower-level concerns. It’s one thing to say you don’t have to worry about backup and security but it means that you now have to make sure that your Cloud provider handles them at an acceptable level. And even on technical grounds, abstractions still leak. Take Google App Engine, for example. In theory you only deal with requests and not even think about the servers that process them (you have no idea how many servers are used). That’s nice, but once a while your Java application gets a DeadlineExceededException. That’s because the GAE platform had to start using a new JVM to serve this request (for example, your traffic is growing or the JVM previously used went down) and it took too long for the application to load in the new JVM, resulting in this loading request being killed. So you, as the developer, have to take special steps to mitigate a problem that originates at a lower level of the stack than you’re supposed to be concerned about.
All in all, the distinction between backward-compatible and forward-compatible Clouds is not a classification (most Cloud environments are a mix). Rather, it’s another mental axis on which to project your Cloud plans. It’s another way to think about the benefits that you expect from your use of the Cloud. Both providers and consumers should understand what they are aiming for on that axis. Hopefully this can help prevent shout matches of the “it’s a bug, no it’s a feature” variety.
[UPDATED 2010/3/5: I now have the quasi-proof that indeed Steve Ballmer stole the idea from my blog. Look at this entry in my HTTP log. This visitor came the evening before Steve’s “Cloud” talk at the University of Washington. I guess I am not the only one to procrastinate until the 11th hour when I have a deadline. Every piece of information in this log entry points at Steve Ballmer. How can it be anyone other than him?
131.107.0.71 - - [03/Mar/2010:23:51:52 -0800] "GET /archives/1198 HTTP/1.0" 200 4820 "http://www.bing.com/search?q=Brilliant+Cloud+Insight" "Mozilla/1.22 (compatible; MSIE 2.0; Microsoft Bob)"
Chuck Shotton recently made a compelling case (“Twitter with a Brain“) for Twitter tools to allow the user to change the protocol endpoint. That is, instead of always going to twitter.com, you can tell your Twitter client to send all requests to myTweetInterceptor.me.com. Why would you do this? You should read his blog entry, but in short his point is that the intermediary can add all kinds of new features that neither the Twitter client nor Twitter itself support. As always in computer science, a new level of decoupling adds opportunities for extensions (and breakage too, of course).
Why just Twitter? This should be true for any client using any protocol. Why not also the APIs for the various Google and Yahoo services? The APIs for the other social networks beyond Twitter? For shopping sites like Amazon and EBay? Etc. And of course to all the various Cloud providers out there. Just because I am using the Amazon EC2 API it doesn’t mean I necessarily want the requests to go straight to Amazon. Client tools should always make the endpoint configurable, period.
Chuck’s examples are about features that the Twitter clients could provide but don’t, so an intermediary would be an easy way to hack support for them (others presumably include modifying the client – if open source -, writing a plug-in for it – it there is such mechanism -, or running a network interceptor on the local client – unless the protocol is encrypted-).
That’s nice and I’d love to see this, but the big deal for me is less with clients and more with third party sites. You know, all these sites that ask for your Twitter login/password. Or those that ask for your GMail/Yahoo account info to retrieve a list of your contacts. I never grant these requests, but I would consider it if they allowed me to tell them what endpoint URL to use. For example, rather than using my Twitter login to go straight to twitter.com, they would use a login/password that I create and talk to twitterIntermediary.vambenepe.com. The requests would be in the exact same shape as what they send today to Twitter, just directed to another URL. There, I could have a proxy that only allows some requests (e.g. “update twitter background image” but not “send update”) and forwards them using my real Twitter credentials. Or, for email accounts, I could have a proxy that allows requests that read my address book but not those that read my mails. The goal here is not to add features, it is to delegate trust in a fine-grained (and audited) manner. This, to me, is the burning need, rather than a 3rd place to implement Twitter lists.
I would probably write these proxies using a PaaS platform like the Google App Engine. Or maybe even Yahoo Pipes. I have long struggled to think of use cases for which Yahoo Pipes hits the sweetspot, and this may well be it. Especially if people write modules to handle specific APIs (e.g. a “Twitter API” module that shows all operations and lets you enable/disable them one by one in a pipe). The one thing missing would be a way for a pipe to keep a log of its invocations, for auditing.
You want access to my email and social network accounts? Give me the ability to filter you requests and you’ll get access. If it’s blind trust you want, I am afraid I have a very limited supply.
The potential user impact of changes (e.g. patches or config changes) made on the Cloud infrastructure (by the Cloud provider) is a sore point in the Cloud value proposition (see Hoff’s take for example). You have no control over patching/config actions taken by the provider, any of which could potentially affect you. In a traditional data center, you can test the various changes on specific applications; you don’t have to apply them at the same time on all servers; and you can even decide to skip some infrastructure patches not relevant to your application (“if it aint’ broken…”). Not so in a Cloud environment, where you may not even know about a change until after the fact. And you have no control over the timing and the roll-out of the patch, so that some of your instances may be running on patched nodes and others may not (good luck with troubleshooting that).
Unfortunately, this is even worse for PaaS than IaaS. Simply because you seat on a lot more infrastructure that is opaque to you. In a IaaS environment, the only thing that can change is the hardware (rarely a cause of problem) and the hypervisor (or equivalent Cloud OS). In a PaaS environment, it’s all that plus whatever flavor of OS and application container is used. Depending on how streamlined this all is (just enough OS/AS versus a traditional deployment), that’s potentially a lot of code and configuration. Troubleshooting is also somewhat easier in a IaaS setup because the error logs are localized (or localizable) to a specific instance. Not necessarily so with PaaS (and even if you could localize the error, you couldn’t guarantee that your troubleshooting test runs on the same node anyway).
In a way, PaaS is squeezed between IaaS and SaaS on this. IaaS gets away with a manageable problem because the opaque infrastructure is not too thick. For SaaS it’s manageable too because the consumer is typically either a human (who is a lot more resilient to change) or a very simple and well-understood interface (e.g. IMAP or some Web services). Contrast this with PaaS where the contract is that of an application container (e.g. JEE, RoR, Django).There are all kinds of subtle behaviors (e.g, timing/ordering issues) that are not part of the contract and can surface after a patch: for example, a bug in the application that was never found because before the patch things always happened in a certain order that the application implicitly – and erroneously – relied on. That’s exactly why you always test your key applications today even if the OS/AS patch should, in theory, not change anything for the application. And it’s not just patches that can do that. For example, network upgrades can introduce timing changes that surface new issues in the application.
And it goes both ways. Just like you can be hurt by the Cloud provider patching things, you can be hurt by them not patching things. What if there is an obscure bug in their infrastructure that only affects your application. First you have to convince them to troubleshoot with you. Then you have to convince them to produce (or get their software vendor to produce) and deploy a patch.
More realistically, narrowly define container types to reduce both the contract and the size of the underlying implementation of each instance. For example, rather than deploying a full JEE+SOA container componentize the application so that each component can deploy in a small container (e.g. a servlet engine, a process management engine, a rule engine, etc). As a result, the interface exposed by each container type can be more easily and fully tested. And because each instance is slimmer, it requires fewer patches over time.
PaaS providers may give their users some amount of visibility and control over this. For example, by announcing upgrades ahead of time, providing updated nodes to test on early and allowing users to specify “freeze” periods where nothing changes (unless an urgent security patch is needed, presumably). Time for a Cloud “refresh” in ITIL/ITSM-land?
Another incarnation of this problem is likely to show up in the form of PaaS certification. We should not assume that just because you use a PaaS you are the developer of the application. Why can’t I license an ISV app that runs on GAE? But then, what does the ISV certify against? A given PaaS provider, e.g. Google? A given version of the PaaS infrastructure (if there is such a thing… Google advertises versions of the GAE SDK, but not of the actual GAE runtime)? Or maybe a given PaaS software stack, e.g. the Oracle/Microsoft/IBM/VMWare/JBoss/etc, meaning that any Cloud provider who uses this software stack is certified?
I have only discussed here changes to the underlying platform that do not change the contract (or at least only introduce backward-compatible changes, i.e. add APIs but don’t remove any). The matter of non-compatible platform updates (and version coexistence) is also a whole other ball of wax, one that comes with echoes of SOA governance discussions (because in PaaS we are talking about pure software contracts, not hardware or hardware-like contracts). Another area in which PaaS has larger challenges than IaaS.
Finally, for an illustration of how a highly focused and specialized container cuts down on the need for config changes, look at this photo from earlier today during the presentation of JRockit Virtual Edition at Oracle Open World. This slide shows (in font size 3, don’t worry you’re not supposed to be able to read), the list of configuration files present on a normal Linux instance, versus a stripped-down (“JeOS”) Linux, versus JRockit VE.
By the way, JRockit VE is very interesting and the environment today is much more favorable than when BEA first did it, but that’s a topic for another post.
I don’t know anyone in Silicon Valley who can code and doesn’t fantasize about writing an accidental killer app. One that gets designed during a long layover in DEN and implemented in a rainy weekend (El Nino is my VC). One that was only supposed to meet the needs of a few friends and is used by half of the world a few months/years later.
By definition, hobbyists only do things that are satisfying. In the rarefied air of Silicon Valley, it also helps if there is a conceivable “upside” to dream about. Platform as a Service (PaaS, e.g. Google App Engine) provides both to software-oriented hobbyist. And make it very cheap (borderline free), which doesn’t hurt.
In a well-crafted PaaS environment, the development process and the result are both satisfying. I am not a Google shrill, but GAE is a fair example. The barrier to entry is very low (the download is less than 10MB and contains all you need to get started). In an hour you have an application running locally. In an hour and 5 minutes you have it deployed and accessible on the web for all. And yet this ease of bootstrap does not come at the cost of too many longer-term limitations (now that the environment has gown a bit from the original limitations and provides scheduled and background jobs). Unlike Yahoo Pipes, for which the first impression is “nifty!” and the second is “gimme a textual representation of my pipe now!”.
Beyond the easy ramp-up, the main source of satisfaction developing in a PaaS environment is that you spend 99% of your time working on the application. Not the OS, not the firewall, not the application container, not the database. Not to mention having to deal with your co-lo provider or the leased line for the servers in the basement. If you are a hobbyist with only a few spare hours per week, that’s a make or break deal. It also means that you have a fighting chance of developing a secure application because you are responsible for a much smaller surface of attack.
Eons ago (in computing time), Visual Basic was the name of the game for these people. More recent was the rise of PHP. It dramatically lowered the barrier to adoption and provided a quick route to a working web application. I know several non-professional developers (e.g. web designers) who are scared of any “normal” programming language but happily write PHP (often of equivalent complexity BTW). Combine this with the wide availability of ISP-managed PHP environments and you get close to what GAE gives you. At the risk of adding to the annoying trend of retroactively cloudifying everything, I think of ISP-hosted PHP as the first generation of PaaS. But it is focused on “show what’s in the DB” scenarios rather than service-centric / mash-up / web 2.0 integration. And even for DB-centric scenarios, by and large PHP coders don’t want to think too much about model and queries (and it shows). I think Google decided to go with Python rather than the easy route of aping the hosted PHP environments in large part to avoid hitting such ceilings down the road. Not surprisingly, PHP support is currently the most requested GAE feature, ahead of Perl and Ruby. Lets see if Google tries to get the PHP community on board or prefers to stay clear of such PaaS legacy (already!).
In the unlikely event that your application catches fire and sees wide adoption (which is not impossible, especially if well integrated in a social network), what are you going to do about it? Keeping in mind two constraints: first, this is a part-time hobby of yours. Second, don’t dream of riches. We are talking about an influx of facebookers or twitterers here, with no intention to pay for anything. But click they will. If you were going to answer: “I get funding and hire a real IT staff” then think again. You most likely won’t get funding for your toy app without revenue potential. And even if you do, by the time you have it it’s too late and people have moved on because you were not there when the spotlight was on you.
With a PaaS-based application you have a fighting chance. If the spike is short enough, you may not even hit the limit of the free quota. If it does, you have the choice of whether you are willing to pay to support the extra traffic or not. No change in code required (though it may be advised anyway, if your app wasn’t architected for efficient scaling – PaaS doesn’t entirely take this off your hands).
That “sudden spike” story is a commonly-invoked use case for EC2. And it’s probably true for a start-up with an IT staff (of at least one full-time person). But despite Amazon’s efforts (and other providers such as RightScale) this type of scaling is something you have to architect for and putting it in place takes away from the time you spend coding application features. It also means that you are responsible for more infrastructure (OS and application container at least). Not to mention that IaaS providers don’t usually offer free resources for limited usage, the way Google does (I suspect 99% of GAE apps never get over this quota). Even if a small EC2 instance is not very expensive (though it adds up over time if you keep it up for that occasional user), the difference between “free” and “cheap” is significant. As an application provider you’d like for this not to be the case with your users, but as a consumer of infrastructure service you’re on the other side of the deal, aren’t you?
There is a reason why suburbia-bound SUVs are advertised crossing mountain streams. The “I could if I wanted to” line has appeal. For the software hobbyist, knowing that your application won’t crash if it happened to meet success (even if only for a couple of days, e.g. the Slashdot effect) is a good feeling (“I could if *they* wanted to”). In truth this occasion is rare (and likely to end up like this), but you are ready for the eventuality. And if there are enough such hobbyists, then statistically some will encounter it.
That last point brings the topic of the PaaS provider’s upside in this. I have read several critical comments arguing that no company will rewrite their application for GAE (true) and that no start-up will write their new code for it either because of the risk of lock-in (also true: “being bought by Google” is not a bad outcome but “has to be bought by Google” is a bad exit strategy). But I think this misses the point of casting such a wide nest and starting with creating a great tool for hobbyists.
Some of these hobbyist applications may actually take off and Google becomes their natural partner/godfather (including managing their user accounts). For example, wouldn’t it be nice for Google if Craigslist or Twitter was running on GAE?
The platform eventually evolves into something that makes sense for start-ups, SMBs and/or enterprises to use. Google works out the kinks with less demanding users first.
A platform to build/run your application is one thing. A way to reach users is another (arguably much more critical). Things like mobile app stores (especially Apple’s of course), Facebook and next generation app stores. But this goes beyond the scope of this post.
Just to be clear, I am not in any way suggesting that PaaS is only for hobbyists. I am just saying that right now it is a great tool for them, the best way for an individual programmer to have fun and have an impact. This doesn’t take away from the value that PaaS will eventually deliver to larger organizations.
Encouraged by hypervisor vendors, the confusion between virtualization and Cloud Computing is rampant. In the industry, the term “virtualization” (and its corollary, “virtual machine”) is used in so many different ways that it has lost all usefulness. For a recent example, read the introduction of this SNIA/OGF white paper (on Cloud Storage) which asserts that “the new technology underlying this is the system virtual machine that allows multiple instances of an operating system and associated applications to run on single physical machine. Delivering this over the network, on demand, is termed Infrastructure as a Service (IaaS)”.
In fact, even IaaS-type Cloud services don’t imply the use of hypervisors.
We need to decouple the Cloud interface/contract (e.g. “what are the types of resources that can I provision on demand? hosts, app servers, storage capacity, app services…”) from the underlying implementation (e.g. “are hypervisors used by the Cloud provider?”). At the risk of spelling out things that may be obvious to many readers of this blog, here is a simplified matrix of Cloud Computing systems, designed to illustrate that all combinations of interface and implementation are possible and in many cases even reasonable.
This is a very common approach these days, both in public Clouds (EC2, Rackspace and presumably at some point the VMWare vCloud Express service providers) and private Clouds (Citrix, Sun, Oracle, Eucalyptus, VMWare…). Basically, you take a bunch of servers, put hypervisors on all of them and make VMs running on these hypervisors available to the Cloud customers.
But despite its predominance, this is not the only path to a Cloud, not even to an IaaS (e.g. “x86 hosts on demand”) Cloud. The following three other scenarios are all valid too.
This is the road SpringSource has been on, first with Cloud Foundry (using AWS EC2 which is based on the Xen hypervisor) and presumably soon on top of VMWare.
Let’s remember that the utility computing vision (before the term fell in desuetude in favor of “cloud”) has been around before x86 hypervisors were so common. Take Loudcloud as an illustration. They were building what is now called a “public Cloud” starting back in 1999 and not using any hypervisor. Just bare metal provisioning and advanced provisioning automation software. Then they sold the hosting part to EDS (now HP) and only kept the software, under the name Opsware (now HP too, incidentally). That software was meant to create what we now call a “private Cloud”. See this old DCML announcement as one example of the Opsware vision. And no hypervisor was harmed in the making of this movie.
And this is not going into finner techno-philosophical points about the different types of hypervisors. Not to mention mainframe LPARs… One could build a hypervisor-free IaaS solution on these.
To some extent, you may even put the “pwned” machines (in a botnet) in this “IaaS with no hypervisor” category (with the small difference that what’s being made available is an x86 with an OS, typically Windows, already installed). If you factor out externalities (like the FBI breaking down your front door at 6:00AM) this approach has claims as the most cost-effective form of Cloud computing available today… Solaris zones are another example of possible foundation for a hypervisor-free IaaS-like offering (here too, with an OS rather than a “raw host” as the interface).
In the public sphere, this corresponds to Google App Engine.
In the private sphere, several companies have built it themselves on top of WebLogic, by adding some level of “on-demand” application provisioning in order to streamline the relationship between the IT group running the servers and the business groups who want to deploy applications on them. Something that one should ideally be able to buy rather than build.
Like most deeply-ingrained confusions, the conflation of virtualization and Cloud Computing won’t be dispelled as much as made irrelevant. The four categories enumerated in this post are a point-in-time view of a continuously evolving system. What may start today as a bundle of a hypervisor, an OS and an app server may become a somewhat monolithic “PaaS engine” over time as the components are more tightly integrated. That “engine” may have memory isolation mechanisms that look a lot like a hypervisor. But it may not be able to host a generic OS. In the same way that whales don’t have fingers and toes and yet they are still very much apparent in their skeleton.
DeltaCloud shares the same goal of translating between different Cloud management protocols but they present their own interface as yet another Cloud REST API/protocol rather than a language-specific toolkit. More along the lines of what UCI is trying to do (not sure what’s up with that project, I recorded my skepticism earlier and am still waiting to be pleasantly surprised).
Of course there are also programming toolkits that are specific to one Cloud provider. They are language-specific wrappers around one Cloud management protocol. AWS protocols (EC2, S3, etc…) represent the most common case, for example amazon-ec2 (a Ruby Gem), Power-EC2Dream (in C# which gives it the tantalizing advantage of being invokable via PowerShell) and typica (for Java). For Clouds beyond AWS, check out the various RightScale Ruby Gems.
The main point of this entry was to list the cross-Cloud development toolkits in the bullet list above. But if you’re in the mood for some pontification you can keep reading.
For some reason, what used to be called “protocols” is often called “APIs” in Cloud settings. Witness the Sun Cloud “API” or the vCloud “API” which only define XML formats for on-the-wire messages. I have never heard of CIM/XML over HTTP, WSDM or WS-Management being referred as APIs though they occupy a very similar place. They are usually considered “protocols”.
It’s a just question of definition whether an on-the-wire protocol (rather than a language-specific set of objects/methods) qualifies as an “Application Programming Interface”. It’s not an “interface” in the Java sense of the term. But I can “program” against it so it could go either way. On this blog I have gone along with the “API” term because that seemed widely used, though in verbal conversations I have tended to stick to “protocol”. One problem with “API” is that it pushes you towards mixing the “what” and the “how” and not respecting the protocol/model dichotomy.
Where is becomes relevant is when you start to see language-specific APIs for Cloud control pop-up as listed above. You now have two classes of things called “API” and it gets a bit confusing. Is it time to bring back the “protocol” term for on-the-wire definitions?
As a developer, whether you’re better off eating your Cloud noodles using chopsticks (on-the-wire protocol definitions) or a fork (language-specific APIs) is an important decision that will stay with you and may come back to bit you (e.g. when the interfaces are versioned). There is a place for both of course, but if we are to learn anything from WS-* it’s that we went way too far in the “give me a java stub” direction. Which doesn’t mean there is no room for them, but be careful how far from the wire semantics you get. It become even trickier when your stub tries not jsut to bridge between XML and Java but also to smooth out the differences between several on-the-wire protocols, as the toolkits above do. The hope, of course, is that there will eventually be enough standardization of on-the-wire protocols to make this a moot point.
Drip drip drip… Is this the sound of your cloud leaking?
It can happen in different ways. See for example this recent research paper, titled “Hey, You, Get Off of My Cloud: Exploring Information Leakage in Third-Party Compute Clouds”. It’s a nice read, especially if you find side channels interesting (I came up with one recently, in a different context).
In the first part of the paper, the authors show how to get your EC2 instance co-located (i.e. running in in the same hypervisor) with the instance you are targeting (the one you want to spy on). Once this is achieved, they describe side channel attacks to glean information from this situation.
This paper got me thinking. I noticed that it does not mention trying to go after disk blocks and memory. I don’t know if they didn’t try or they tried and were defeated.
For disk blocks (the most obvious attack vector), Amazon is no dummy and their “proprietary disk virtualization layer automatically wipes every block of storage used by the customer, and guarantees that one customer’s data is never exposed to another” as explained in the AWS Security Whitepaper. In fact, they are so confident of this that they don’t even bother forbidding block-based recovery attempts in the AWS customer agreement (they seem mostly concerned about attacks that are not specific to hypervisor environments, like port scanning or network-based DOS). I took this as an invitation to verify their claims, so I launched a few Linux/ext3 and Windows/NTFS instances, attached a couple of EBS volumes to them and ran off-the-shelf file recovery tools. Sure enough, nothing was found on /dev/sda2 (the empty 150GB partition of local storage that comes with each instance) or on the EBS volumes. They are not bluffing.
On the other hand, there were plenty of recoverable files on /dev/sda1. Here is what a Foremost scan returned on two instances (both of them created from public Fedora AMIs).
These are blocks in the AMI itself, not blocks that were left on the volumes on which the AMI was installed. In other words, all instances built from the same AMI will provide the exact same recoverable files. The C: drive of the Windows instance also had some recoverable files. Not surprisingly they were Windows setup files.
Again, kudos to Amazon, but I also wonder if this feature opens a possible DOS approach on AWS: it doesn’t cost me much to create a 1TB EBS volume and to destroy it seconds later. But for Amazon, that’s a lot of blocks to wipe. I wonder how many such instantaneous create/delete actions on large EBS volumes it would take to put a large chunk of AWS storage capacity in the “unavailable – pending wipe” state… That’s assuming that they proactively wipe all the physical blocks. If instead the wipe is virtual (their virtualization layer returns zero as the value for any free block, no matter what the physical value of the block) then this attack wouldn’t work. Or maybe they keep track of the blocks that were written and only wipe these.
Then there is the RAM. The AWS security paper tells us that the physical RAM is kept separated between instances (presumably they don’t use ballooning or the more ambitious Xen Transcendent Memory). But they don’t say anything about what happens when a new instance gets hold of the RAM of a terminated instance.
Amazon probably makes sure the RAM is reset, as the disk blocks are. But what about your private Cloud infrastructure? While the prospect of such Cloud leakage is most terrifying in a public cloud scenario (anyone could make use of it to go after you), in practice I suspect that these attack vectors are currently a lot more exploitable in the various “private clouds” out there. And that for many of these private clouds you don’t need to resort to the exotic side channels described in the “get off of my cloud” paper. Amazon has been around the block (no pun intended) a few times, but not all the private cloud frameworks out there have.
One possible conclusion is that you want to make sure that your cloud vendor does more than writing scripts to orchestrate invocations of the hypervisor APIs. They need to understand the storage, computing and networking infrastructure in details. There is a messy physical world under your clean shinny virtual world. They need to know how to think about security at the system level.
Another one is that this is a mostly an issue for hypervisor-based utility computing and a possible trump card for higher level of virtualization, e.g. PaaS. The attacks described in the paper (as well as block-based file recovery) would not work on Google App Engine. What does co-residency mean in a world where subsequent requests to the same application could hit any machine (though in practice it’s unlikely to be so random)? You don’t get “deployed” to the same host as your intended victim. At best you happen to have a few requests executed while a few requests of your target run on the same physical machine. It’s a lot harder to exploit. More importantly, the attack surface is much more restrained. No direct memory access, no low-level scheduler data, no filesystem… The OS to hardware interface that hypervisors emulate was meant to let the OS control the hardware. The GAE interface/SDK, on the other hand, was meant to give the application just enough capabilities to perform its task, in a way that is as removed from the hardware as possible. Of course there is still an underlying physical reality in the GAE case and there are sure to be some leaks there too. But the small attack surface makes them a lot harder to exploit.
I am probably taking things a bit too personally, but I feel like I just successfully guilt-tripped the Google App Engine (GAE) team. Just last night I was complaining that they were teasing me (supporting urllib2, but an older version, without timeout support). And tonight I noticed a new post on their blog, announcing the end of these “High CPU Requests” that have been the bane of my GAE experience.
The reason why I was looking for timeout support in the first place is to avoid generating these dreaded “High CPU Requests” that quickly result in your application being disabled. It’s all explained here and here.
But now that they are gone, I don’t need timeouts anymore. Around 11:00PM Pacific (on Thursday 2/12) I restarted my application. All it does is create 5 new simple entities in the datastore, one every second (it sleeps for one second between each entry). Then it spawns its successor, which will do the same thing, ad vitam aeternam. The application’s name, “rere”, is short for “request relay”, the pattern used to emulate a long running process. The default page for the app (available here) just returns a list of the 30 last entities created. The point is to illustrate that a single original request spawned an ever-lasting computing task on GAE.
# The mere existence of an instance of this class in the DB means that the relay has to stop.
# "App cannot fetch the same URL as the one used for the request" error.
One thing I had to change from the earlier version (written using version 1.1.0 of the GAE SDK) is that urlfetch now returns an error if your app tries to invoke itself at the same URL (“App cannot fetch the same URL as the one used for the request”). So I have to alternate between http://localhost/start and http://localhost/start2, both of which are mapped to the same handler. This was added sometimes between SDK 1.1.0 and SDK 1.1.9. If it is aimed at preventing the kind of batton-passing that I am doing, it is pretty unefficient considering how easy it is to circumvent.
Which shows that 1056 successive requests have participated in the relay (the last one just happened, at 09:02:15 UTC which is 1:02AM Pacific).
Hopefully it will still be running when I wake up tomorrow.
[UPDATED 2009/2/13, 9:08AM Pacific: It’s alive!
Version 1.1.9 of the Google App Engine (GAE) SDK was released earlier this week. The first item in the announcement covers the big news, that developers can “use the Python standard libraries urllib, urllib2 or httplib to make HTTP requests”. My first thought reading this was that I was finally going to be able to use timeouts on outgoing HTTP requests. I care about this because my earlier attempt to emulate a long-running process in GAE was stymied by the GAE quota system, something I think I can work around if I can timeout a request once it has spawned its successor (more precisely, once it has spawned the successor of the incoming request that created the new outgoing request).
I got the new SDK last evening and moved the code from using urlfetch to using urllib2 (with timeout). On my local machine it seems to work, but the quota system (that I am trying to finesse) doesn’t run in the SDK. So the only real test happens once you deploy the app in the Google environment. Which is when I realized that GAE uses Python 2.5.2 and that the timeout parameter in urllib2 (and httplib) came with 2.6. Slap.
I was especially disappointed because the links for urllib2 and httplib in the GAE 1.1.9 SDK announcement take us to the Python 2.6.1 documentation. The timeout parameter is right there at the top of these pages, staring at me. It would be more accurate for this announcement to point to the 2.5.2 documentation (here it is for urllib2 and httplib).
It doesn’t really matter of course because this is just a toy project. And a real scheduler seems to be in the works (see this pre-announcement and this work in progress). I just do this as a fun way to get a glimpse of what it takes to turn existing infrastructure into a *aaS product, something that is going on in different ways in many places these days. Linux wasn’t created to run on something else than hardware. Xen wasn’t created to support EC2. Python wasn’t created to support GAE.
And GAE wasn’t created to support long-running processes. But I haven’t given up.
I am writing a quick post on GAE today because the Google team just published a blog entry that explains why my attempts to create a long-running process in GAE resulted in my application being disabled for having too many “high CPU requests”, a concept that was never documented in the quota system.
Google is now offering a “quota detailed dashboard” and, as this screen capture shows, it tells us that you only get 60 “high CPU requests” and that they replenish at a rate of 2 per minutes. So at least now I know.
As promised to Felix here is the code that shows how to give extra breathing room to Google App Engine (GAE) requests that may otherwise be killed for taking too long to complete. The approach is similar to the one previously described. But rather than trying to emulate a long-running process, I am simply allowing a request to spread its work over a handful of invocations, thus getting several 9 seconds slots (since this seems to be how much time GAE gives you per request right now).
If all your requests need this then you are going to run into the same “high CPU requests have a small quota, and if you exceed this quota, your app will be temporarily disabled” problem seen in the previous experiment. But if 90% of your requests complete in a normal time and only 10% of the requests need more time, then this approach can help prevent your users from getting an error for 1 out of every 10 requests. And you should fly under the radar of the GAE resource cop.
The way it works is simply that if your request is interrupted for having run too long the client gets a redirect to a new instance of the same handler. Because the code saves its results incrementally in the datastore, the new instance can build on the work of the previous one.
This specific example retrieves the ubuntu-8.04-server-i386.jigdo file (98K) from a handful of Ubuntu mirrors and returns the average/min/max download times (without checking if the transfer was successful or not). I also had to add a 1 second sleep after each fetch in order to trigger the DeadlineExceededError because the fetch operations go too quickly when running on GAE rather than my machine (I guess Google has better connectivity than my mediocre AT&T-provided DSL line, who would have thought).
This request used a high amount of CPU, and was roughly 28.5 times over the average request CPU limit.
High CPU requests have a small quota, and if you exceed this quota, your app will be temporarily disabled.
This request used a high amount of CPU, and was roughly 13.4 times over the average request CPU limit.
I believe we can optimize the performance by taking advantage of the fact that successive requests are likely (but not guaranteed) to hit the same instance, allowing global variables to be re-used rather than always going to the datastore. My code is a proof of concept, not an optimized implementation.