Wednesday, December 28, 2016

Dynamodb restore using datapipeline with newer emr version

68 comments
We faced an interesting issue here.

We had a task to restore dynamodb from backup stored in s3. We took the backup using AWS pipeline. But when we started restoring, which we had been doing for a long time, we found the cluster was not able to provisioned and it was failing on bootstraping.
Digging more into it we found the AWS Dynamodb import template of datapipeline use default subnet under default VPC. Now, we don't have Internet Gateway for this VPC. Since, EMR needs IG to work successfully, this pipeline was failing with "internal error"

We started talking with AWS support. After some discussion, we changed our private subnet. Again the EMR provisioning was failing as the template to restore dynamodb that provided by AWS uses AMI Version 3.9.0 which does not support private subnet.

So we decided to change the AMI Version 3.9 to release label "emr-4.5.0" which we have been using for all our EMR so far. Again we failed with error:
Unable to create resource for @EmrClusterForLoad_2016-12-28T20:33:21 due to: The supplied bootstrap action(s): 'bootstrap-action.7237c1e1-31de-4c02-ae68-c546dd581732' are not supported by release 'emr-4.5.0'. (Service: AmazonElasticMapReduce; Status Code: 400; Error Code: ValidationException; Request ID: e8be350e-cd3c-11e6-8e60-cb10b4c3228c)

That is, the template script provided by AWS does not support emr release label 4.5.0. To overcome the problem, we had to modify EmrCluster bootstrap action in pipeline definition which was:
s3://#{myDDBRegion}.elasticmapreduce/bootstrap-actions/configure-hadoop, --mapred-key-value,mapreduce.map.speculative=false

This was only supported by AMI 3.9.0. For release label emr-4.5.0, it should be added as configuration properties as follows:
--
        {
            "configuration": {
                "ref": "EmrConfigurationId_XXWNE"
            },
            "releaseLabel": "emr-4.5.0",
            "type": "EmrCluster",
            ...
       },
       {
            "property": {
                "ref": "PropertyId_3ghq7"
            },
            "type": "EmrConfiguration",
            "id": "EmrConfigurationId_XXWNE",
            "classification": "mapred-site",
            "name": "DefaultEmrConfiguration1"
        },
        {
            "key": "mapreduce.map.speculative",
            "type": "Property",
            "id": "PropertyId_3ghq7",
            "value": "false",
            "name": "DefaultProperty1"
        },
--

Now, we exported the pipeline definition and added the above configuration. The final pipeline definition was looking like this:

{
  "objects": [
    {
      "property": [
        {
          "ref": "PropertyId_3ghq7"
        }
      ],
      "name": "DefaultEmrConfiguration1",
      "id": "EmrConfigurationId_XXWNE",
      "type": "EmrConfiguration",
      "classification": "mapred-site"
    },
    {
      "output": {
        "ref": "DDBDestinationTable"
      },
      "input": {
        "ref": "S3InputDataNode"
      },
      "maximumRetries": "1",
      "name": "TableLoadActivity",
      "step": "s3://dynamodb-emr-#{myDDBRegion}/emr-ddb-storage-handler/2.1.0/emr-ddb-2.1.0.jar,org.apache.hadoop.dynamodb.tools.DynamoDbImport,#{input.directoryPath},#{output.tableName},#{output.writeThroughputPercent}",
      "runsOn": {
        "ref": "EmrClusterForLoad"
      },
      "id": "TableLoadActivity",
      "type": "EmrActivity",
      "resizeClusterBeforeRunning": "true"
    },
    {
      "subnetId": "subnet-xxxxxxx",
      "name": "EmrClusterForLoad",
      "coreInstanceCount": "1",
      "coreInstanceType": "m3.xlarge",
      "releaseLabel": "emr-4.5.0",
      "id": "EmrClusterForLoad",
      "masterInstanceType": "m3.xlarge",
      "region": "#{myDDBRegion}",
      "type": "EmrCluster",
      "terminateAfter": "23 Hours",
      "configuration": {
                "ref": "EmrConfigurationId_XXWNE"
            }
    },
    {
      "failureAndRerunMode": "CASCADE",
      "resourceRole": "PSS-BDP-QA-DataPipelineDefaultResourceRole",
      "pipelineLogUri": "s3://pss-bdp-qa-logfiles/datapipeline-logs/PSS-BDP-SQA-Dynamodb-Import-1/",
      "role": "PSS-BDP-DataPipelineDefaultRole",
      "scheduleType": "ONDEMAND",
      "name": "Default",
      "id": "Default"
    },
    {
      "writeThroughputPercent": "#{myDDBWriteThroughputRatio}",
      "name": "DDBDestinationTable",
      "id": "DDBDestinationTable",
      "type": "DynamoDBDataNode",
      "tableName": "#{myDDBTableName}"
    },
    {
      "directoryPath": "#{myInputS3Loc}",
      "name": "S3InputDataNode",
      "id": "S3InputDataNode",
      "type": "S3DataNode"
    },
    {
        "key": "mapreduce.map.speculative",
        "type": "Property",
        "id": "PropertyId_3ghq7",
        "value": "false",
        "name": "DefaultProperty1"
    }
  ],
  "parameters": [
    {
      "description": "Input S3 folder",
      "id": "myInputS3Loc",
      "type": "AWS::S3::ObjectKey"
    },
    {
      "description": "Target DynamoDB table name",
      "id": "myDDBTableName",
      "type": "String"
    },
    {
      "default": "0.25",
      "watermark": "Enter value between 0.1-1.0",
      "description": "DynamoDB write throughput ratio",
      "id": "myDDBWriteThroughputRatio",
      "type": "Double"
    },
    {
      "default": "us-east-1",
      "watermark": "us-east-1",
      "description": "Region of the DynamoDB table",
      "id": "myDDBRegion",
      "type": "String"
    }
  ],
  "values": {
    "myDDBRegion": "us-east-1",
    "myDDBTableName": "TABLE_TEST",
    "myDDBWriteThroughputRatio": "1",
    "myInputS3Loc": "s3://my-dynamobackup/TABLE_TEST_201609/2016-12-22-22-55-57"
  }
}

68 Responses so far

  1. piyagupta says:

    Bangalore Escorts provides escort call girls by the escort agency in Bangalore. We have selected the best high profile call girls in Bangalore. Visit us www.piyagupta.com/
    South Bangalore Escorts
    Marathahalli Escorts
    Electronic City Escorts
    Hebbal Escorts
    Mg road Escorts
    Ulsoor Escorts
    Ub city Escorts
    Nandi hills Escorts
    Malleswaram Escorts

  2. Sexy escorts in Chennai with the best reviews. Gorgeous young busty escorts Chennai incall/outcall, our escort girls provide a 1st class experience for gentlemen. www.chennai-escort.com/
    Anna Nagar Escorts
    Chetput Escorts
    Guindy Escorts
    KK Nagar Escorts
    Saidapet Escorts
    t nagar Escorts
    Villivakkam Escorts
    Adyar Escorts
    Ashok Nagar Escorts

  3. Shoppa.in says:


    If you searching for high-class independent escort girls in Hyderabad I and Hyderabad to full enjoy with Hyderabad call girls top class independent escort girls in Hyderabad

    Call Girls in Hyderabad
    Call Girl in Chandigarh

  4. You can book call girls in Guwahati or escorts in Guwahati for incall facility or outcall Facility
    as and when required.
    Escort Services in Guwahati
    Guwahati Escort Service

  5. Escort girls in Delhi make independent Brahma; Delhi call girls are always ready to serve you for interested people. Call girls in Delhi have been placed in many categories. Surely we are available according to you, you can come on our website and contact us. Delhi Escort Service

  6. It feels astonishing and satisfying to have an energetic young lady in your arms while longing for some warmth during desolate winter's evenings. You can pick an energetic school young lady escort in Mumbai from our gigantic assortment and request that she accomplish something astonishing that could please you past your minds. visit my blog :-
    Call Girls Trident Hotel Bandra Kurla &&
    Call Girls Hotel Hiltop &&
    Call Girls ITC Grand Central, A Luxury Collection Hotel &&
    Call Girls ITC Maratha, a Luxury Collection Hotel &&
    Call Girls InterContinental Mumbai Central &&
    Call Girls Hotel Bawa Continental &&
    Call Girls JW Marriott Mumbai Sahar &&
    Call Girls The Leela Mumbai &&
    Call Girls The Lalit Mumbai &&
    Call Girls Peninsula Redpine &&

  7. Looking for a Jaipur Call Girl? Verified Jaipur call girls available for bookings on our adult entertainment, hot, sexy & Beutifull escorts in Jaipur availble now. www.jaipurescortscity.com
    Udaipur Escorts
    Pushkar Escorts
    Lucknow Escorts
    Jaisalmer Escorts
    Ajmer Escorts
    Bikaner Escorts
    Dholpur Escorts
    Kota Escorts
    Jodhpur Escorts

  8. Looking for a Jaipur Call Girl? Verified Jaipur call girls available for bookings on our adult entertainment, hot, sexy & Beutifull escorts in Jaipur availble now. www.jaipurescortscity.com
    Udaipur Escorts
    Pushkar Escorts
    Lucknow Escorts
    Jaisalmer Escorts
    Ajmer Escorts
    Bikaner Escorts
    Dholpur Escorts
    Kota Escorts
    Jodhpur Escorts

  9. Book Chennai Escorts Service for your inner soul satisfaction. Our Female Escorts girls in Chennai make your life amazing. Book your hottest Call Girls in Chennai at - www.chennai-escort.com.
    Anna Nagar Escorts
    Chetput Escorts
    Guindy Escorts
    KK Nagar Escorts
    Saidapet Escorts
    t nagar Escorts
    Villivakkam Escorts
    Adyar Escorts
    Ashok Nagar Escorts

  10. 在美国赌博目录部分,您会发现 บริการการพนันในสหราชอาณาจักร | การพนันสาวยูโร これは最も信頼性の다이짱역 最大のディレクトリの1つです。 ไดเรกทอรีการพนันในยุโรป எல்லா உள்ளடக்கமும் படங்களும் உண்மையான புகைப்படங்களுடன் தொடர்ந்து மதிப்பாய்வு செய்யப்பட்டு וועט זיין דערהייַנטיקט. דאָס קאַנפערמז די אַדס ಎಲ್ಲಾ ಪಂತಗಳು ಯಾವಾಗಲೂ ನವೀಕೃತವಾಗಿರುತ್ತವೆ ಮತ್ತು ನಿಮಗೆ ಉತ್ತಮ ಅನುಭವವನ್ನು ನೀಡುತ್ತದೆ.ನಿಮ್ಮ ಖಾತೆಗೆ ಹಣವನ್ನು ಒದಗಿಸಲು ಇದನ್ನು ಬಳಸಬಹುದು บริการคาสิโนสล็อตของสหรัฐอเมริกา као и за даљу посету: บริการคาสิโนที่ดีที่สุด ในสหรัฐอเมริกา ไวอากร้าที่ดีที่สุดในอินเดีย | таблетка виагры

  11. Enjoy the Passionate service of Hyderabad Escorts. Premium Escorts in Hyderabad for Utmost Entertainment. Truly secured doorway to reach the lovely Hyderabad escort service profiles. Visit now - www.hyderabadescortsagency.co.in
    Hyderabad Escorts
    Hyderabad Escort
    Independent escorts in Hyderabad
    Hyderabad independent escorts
    Hyderabad escorts service
    Hyderabad escort services
    Escorts services in Hyderabad
    Hyderabad escort model
    High class escorts in Hyderabad
    Hyderabad escort models

  12. We provide European Escorts, Russian Escorts in Hyderabad and avail you Erotic massage services at your hotel. For booking call at - www.hyderabadescortsagency.co.in ​right now.
    Hyderabad Escorts
    Banjara Hills Escorts
    Charminar Escorts
    Gachibowli Escorts
    Hi Tech City Escorts
    Jubilee Hills Escorts
    Madhapur Escorts
    Mallapur Escorts

  13. Mumbai Escorts is the best partners for all types of travellers. With various nightlife activities and several options to fulfill their needs, they make great choices for night parties and social events.
    Punjabi Girls Escorts In Bandra ###
    Call Girls In Thane ###
    High Profile Escorts Juhu ###
    Russian Thane Escorts ###
    High Profile Call Girls Andheri ###
    Navi Mumbai Call Girls ###
    Russian Call Girls Navi Mumbai ###
    Santa Cruz Escorts ###
    Call Girls in Goregaon ###

  14. Experience is very important because in this field there are lots of scams and you should always opt for a company that has a lot of experience. Russian Call Girls Chennai with this kind of background would know everything about women and they can easily entice men.
    Celebrity Model Escorts In Chennai &&
    Call Girls Service Chennai &&
    High Profile Chennai Escorts Service &&
    T Nagar Call Girls &&
    Anna Nagar Call Girls &&
    Guindy Call Girls &&
    Adyar Call Girls &&
    Teynampet Call Girls &&
    Egmore Call Girls &&

  15. Looking for escort service or call girls in Hyderabad, just give us a secret moments unforgetable, enjoy 30% discount too, watch Photos. contact now;-https://bit.ly/3yN7pYw
    Secunderabad Escorts
    Hyderguda Escorts
    Lb-Nagar Escorts
    Lingampally Escorts
    Malakpet-Extension Escorts
    Mehdipatnam Escorts
    Moosapet Escorts
    Pathargatti Escorts
    Qutubullapur Escorts

  16. Goa has consistently been a fascination for some, who come to goa for work and different purposes. Goa remains at the bank of the ocean. This thin gated section is encircled by a thick timberland and private properties. The Goa Escorts works on a 24 hour premise and has a few get and drop-off objections in and around goa.
    cavelossim beach call girls
    []
    colva beach call girls
    []
    mandrem beach call girls
    []
    morjim beach call girls
    []
    betalbatim beach call girls
    []
    palolem beach call girls
    []
    panaji call girls
    []
    ponda call girls
    []
    vasco da gama call girls
    []

  17. Monica says:

    The absolute initial step is to push forward to ij.start.cannon This is the authority site of Canon and a safe stage from where you can basically download the Canon Printer Drivers. So to start, open a viable and amazingly ensured internet browser. Type ij.start.cannon accurately in the location bar and by squeezing the Enter key, you will in a flash be taken to the landing page of Canon.

  18. quickbooks error code 15106 is generally when the client is endeavoring to download the QuickBooks work area refreshes. You will get an error message which you can't deliver the refreshed programming.

  19. pogo games not loading


    Check for correct User Name or Screen Name. Most of the time we rely on the browser’s auto-complete feature and select wrongly entered screen name in the past.
    Passwords are case sensitive. Therefore, always check the letter’s cases of the entered password. If possible, type the password as you use in a notepad and then copy and paste the letters in the password field. Below is a list of things you should always check before entering into troubleshooting.
    1. If any of your family member shares the same account, ask them if they might have changed the screen name or password.
    2. Look into your email inbox to get the actual screen name when you had registered with Pogo
    3. If you find any point relevant to your case, you can go to the article explaining how to reset password of your Pogo account and follow all the steps carefully to reset your pogo account password.

    For more detailed information visit:
    https://pogo-supportcenter.com/2020/04/01/fix-pogo-sign-in-issue/

    [url=https://pogo-supportcenter.com/2020/04/01/fix-pogo-sign-in-issue/]pogo sign in problem[/url]

    pogo.com/sign in problems

  20. HP Wireless Printer Setup

    Connecting a printer over a USB cable is pretty easy. But connecting a wireless printer can become very challenging quite often. Follow the HP wireless printer setup process to install it in no time.

    1. Take the printer out of the box and remove all tapes from it before starting.
    2. Plug in the power cord, turn on the printer and install the print cartridges. Allow your printer to complete its startup routine, which includes printing of an alignment page (if applicable).
    3. Go to the manufacturer’s website and download drivers for your printer model.
    4. Install the drivers and follow the instructions during hp wireless printer setup process.
    5. In the meantime, go to priter’s control panel and use wireless printer setup wizard.
    6. Identify your wifi network and connect the printer to it using its correct credential.
    7. Complete the wireless printer setup process by connecting to your printer model as it would appear during installation.

    For more detailed information visit: https://mchelperprintersupport.com/hp-wireless-printer-setup/

    [url=https://mchelperprintersupport.com/hp-wireless-printer-setup/]hp wireless printer setup[/url]

Leave a Reply

Labels