The Curious Engineer

All blogs / Databases

Azure Data Lake & Azure HDInsight Blog

learn.microsoft.com/en-us/archive/blogs/azuredatalake

The official blog for the Azure Data Lake services - Azure Data Lake Analytics, Azure Data Lake Store and Azure HDInsight

DatabasesDistributed systems / cloudDev tools / editors / IDEsMicrosoft blogsMSDN archive

’15’16’17’18’19’20’21’22’23’24’25’26
Posts
134 from 18 Nov 2015 to 31 Jul 2018
Posting
archived, read-only
Kind
company blog
Feed
https://learn.microsoft.com/en-us/archive/blogs/azuredatalake/feed.xml

2018 14

  1. Introducing U-SQL database projects – U-SQL database development and deployment made easy (public preview)
  2. Build hybrid cloud analytics solutions with ADLA Task in SSIS
  3. Easier Azure Data Lake Store management: alerts for folders and files.
  4. Process more files than ever and use Parquet with Azure Data Lake Analytics
  5. Azure Data Lake Analytics and U-SQL Spring 2018 Updates: Parquet support, small files, dynamic output, fast file sets, and much more!
  6. Get started with U-SQL: It’s easy!
  7. Keeping Data Lake Costs Under Control: Creating Alerts for AUs Usage Thresholds.
  8. Simple Trick to Stay on top of your Azure Data Lake: Create Alerts using Log Analytics
  9. Using the first job run to optimize subsequent runs with Azure Data Lake job AU analyzer
  10. Delivering Consistency and Accuracy: Improvements for Azure Data Lake Store Reporting
  11. Struggling to get insights for your Azure Data Lake Store? Azure Log Analytics can help!
  12. From unstructured data to dashboard with Azure Data Factory and Azure Data Lake
  13. Debugging Azure Data Lake Job Failures Made Easy (part 2) - Efficiently troubleshoot anomalies in recurring jobs
  14. How to Save Money and Control Costs with Azure Data Lake Analytics

2017 65

  1. Debugging Azure Data Lake Job Failures Made Easy (part 1) - Debug U-SQL job failure of C# custom code
  2. Ad-hoc query support in Azure Data Lake Tools for Visual Studio
  3. Run your PySpark Interactive Query and batch Job in Visual Studio Code
  4. Getting new insights into your usage of Data Lake Analytics
  5. Find your U-SQL jobs in Azure Data Lake Analytics with one click.
  6. Organize your pipeline and recurring jobs easily with Data Lake Analytics (part 2)
  7. Continuous integration made easy with MSBuild support for U-SQL (preview)
  8. Simple database copies with the Azure Data Lake Database Export Wizard
  9. Organize your pipeline and recurring jobs easily with Data Lake Analytics (part 1)
  10. Azure Data Lake Tools for Visual Studio Code (VSCode) October Updates
  11. Azure Data Lake Analytics and U-SQL Summer 2017 Updates: Introducing GZip on OUTPUT, Catalog Views, Major updates to Cognitive Libraries, Tool support to create your EXTRACT statement and much more!
  12. Managing Pipeline & Recurring Jobs in Azure Data Lake Analytics Made Easy
  13. Directly store streaming data into Azure Data Lake with Azure Event Hubs Capture Provider
  14. U-SQL Tip: Generating ranges of numbers and dates
  15. Running XGBoost on Azure HDInsight
  16. Create U-SQL EXTRACT Script Automatically
  17. Analyze data in Azure Data Lake Store using familiar-and-powerful Excel 2016
  18. Azure Data Lake Tools for Visual Studio Code (VSCode) July updates
  19. Solving the problem of "Problem with the SSL CA cert (path? access rights?)" for R server on HDInsight
  20. Webinar with Talena: Migrate open source big data applications to HDInsight, add backup & restore to your existing apps running on Apache Hadoop & Spark
  21. Run H2O.ai in R on Azure HDInsight
  22. Microsoft R Server 9.1 on HDInsight is available!
  23. Managing Your Azure Data Lake Analytics Compute Resources (Job-level Policy)
  24. Managing Your Azure Data Lake Analytics Compute Resources (Account-level Policy)
  25. Managing Your Azure Data Lake Analytics Compute Resources (Overview)
  26. Announcing Microsoft Machine Learning Library for Apache Spark
  27. HDInsight tools for IntelliJ May updates
  28. HDInsight : BUILD Hive Lab is available now
  29. Azure HDInsight: How to run Presto in one simple step and query across data sources such as Cosmos DB, SQL DB & Hive
  30. Webinar with Datameer: Modern Data Preparation, Analytics and Insights - Key ingredients for an ML- and AI-ready organization
  31. Azure Data Lake Tools for Visual Studio Code (VSCode) General Availability
  32. Using Jupyter notebooks and Pandas with Azure Data Lake Store
  33. SCP.Net with HDInsight Linux Storm clusters
  34. HDInsight tools for IntelliJ & Eclipse April Updates
  35. Azure Data Lake U-SQL April 25 2017 Updates: Introducing Packages, UNPIVOT INCLUDE NULLS, fast file set preview flag, R extension returns dataframes, exporting your cluster database with sample data to your local run and more!
  36. Exposing Hive!
  37. Cloudera clusters now run with Azure Data Lake Store
  38. Use H2O.ai on Azure HDInsight
  39. Azure Data Factory makes it even easier and convenient to uncover insights from data when using Data Lake Store with SQL Data Warehouse
  40. Azure HDInsight 3.6 - Five things that will make a data developer happy
  41. Hive Metastore in HDInsight –Tips, Tricks & Best Practices
  42. How to use BigDL on Apache Spark for Azure HDInsight
  43. Azure Data Lake U-SQL March 9 2017 Updates: Deprecations turn into errors, PIVOT/UNPIVOT, cross ADLS account U-SQL catalog sharing, nuget packages and more!
  44. Using Custom Python Libraries with U-SQL
  45. Analyze your data in ADLS with more assurance with the recently GA'd Power BI Desktop connector
  46. How WebHCat Works and How to Debug (Part 2)
  47. How WebHCat Works and How to Debug (Part 1)
  48. Azure Data Lake Tools for VSCode (Preview) - March Update
  49. Garbage Collection and its performance impact
  50. Wiring your older Hadoop clusters to access Azure Data Lake Store
  51. Restarting Storm EventHub Topology on a new cluster
  52. Using Oozie SLA on HDInsight clusters
  53. Ingest data into Azure Data Lake Store with StreamSets Data Collector
  54. Making Azure Data Lake Store the default file system for Hadoop
  55. Enabling U-SQL Advanced Analytics for Local Execution
  56. Connecting your own Hadoop or Spark to Azure Data Lake Store
  57. Building advanced analytical solutions faster using Dataiku DSS on HDInsight
  58. HDinsight - How to perform Bulk Load with Phoenix ?
  59. Uncover insights rapidly from petabytes of data in Azure Data Lake Store with SQL Data Warehouse PolyBase support
  60. Distributed Deep Learning on HDInsight with Caffe on Spark
  61. U-SQL Deprecation Update: Migration of Data Source Credentials and Removal of CREATE CREDENTIAL, ALTER CREDENTIAL and DROP CREDENTIAL
  62. U-SQL Deprecation notice: PARTITION BY BUCKET will be removed
  63. Introducing: Microsoft Azure Data Lake Tools for Visual Studio Code
  64. HDInsight tools for IntelliJ & Eclipse December Updates
  65. Spark Job Submission on HDInsight 101

2016 50

  1. Cornell Lab of Ornithology Improves Machine Learning Workflow with Azure HDInsight
  2. Introducing: Interactive Hive cluster using LLAP (Long Live and Process)
  3. Spark Debugging on HDInsight 101
  4. Problems with new File Set (update 2016-12-14 - 16:30 PST)
  5. Introducing Python SDKs for Data Lake Store & Analytics
  6. U-SQL Advanced Analytics: Introducing Cognitive scenarios for Text and Imaging
  7. U-SQL Advanced Analytics: Introducing Python Extensions for U-SQL
  8. Apache HBase/Phoenix - Tips , Tricks & Best Practices in HDInsight
  9. Azure Data Lake Store is now generally available
  10. Preview: Azure Data Lake Tools for Visual Studio Code
  11. Executing Spark SQL Queries using dotnet ODBC driver
  12. OozieBot: Automated Oozie Workflow and Coordinator Generation
  13. Azure Data Lake U-SQL October 2016 Updates: Deprecations turn into errors, sampling is live, sharing catalog objects across ADLA accounts, outputting headers and more!
  14. Using Cask Data Application Platform on Azure HDInsight
  15. Azure Data Lake U-SQL September 2016 Updates: OUTER UNION, Set operations by name, FILE and PARTITION intrinsic functions and more!
  16. Understanding the Data Lake Analytics Unit
  17. Azure Data Lake Analytics: More Compute and More Control
  18. Experience Updates to the Azure Data Lake Store and Analytics Portal
  19. HDInsight HBase: How to Improve HBase cluster restart time by Flushing tables?
  20. Getting started with Azure Data Lake Analytics and Store has never been faster!
  21. HDInsight HBase: 9 things you must do to get great HBase performance
  22. HDInsight -New self-paced trainings and labs
  23. How to register U-SQL Assemblies in your U-SQL Catalog
  24. HDInsight:- Attach additional Azure storage accounts to the cluster
  25. Introducing Image Processing in U-SQL
  26. Rapid Big Data Prototyping with Microsoft R Server on Apache Spark: Context Switching & Spark Tuning
  27. Optimizing Apache HBase for Cloud Storage in Microsoft Azure HDInsight
  28. Azure Data Lake U-SQL August 1st 2016 Updates: ACLs on Databases, Skipping Header Rows, Sampling and more!
  29. Introducing File and Folder ACLs for Azure Data Lake Store
  30. HDinsight - How to use Spark-HBase connector?
  31. Azure Data Lake U-SQL July Updates
  32. Partial Caching of DataFrame by Vertical and Horizontal Partitioning
  33. HDInsight tool in Azure Toolkit for Eclipse is GA!
  34. How do I combine overlapping ranges using U-SQL? Introducing U-SQL Reducer UDOs
  35. Azure Data Lake Analytics: Greater flexibility with assigning Parallelism to U-SQL Jobs
  36. Appending an Index Column to Distributed DataFrame based on another Column with Non-unique Entries
  37. HDInsight Tool for IntelliJ is GA!
  38. Leveraging Azure Data Lake Partitioning to Recalculate Previously Processed Days
  39. April Updates to Azure Data Lake Analytics and Azure Data Lake Store
  40. Debugging U-SQL Error E_RUNTIME_USER_EXTRACT_UNEXPECTED_NUMBER_COLUMNS: Unexpected number of columns in input record
  41. HDInsight jobs troubleshooting
  42. HDInsight Hive job workload
  43. HDInsight Hive workload under covers
  44. HDInsight under covers
  45. Saving Spark Streaming Metrics to PowerBI
  46. Saving Spark Resilient Distributed Dataset (RDD) To PowerBI
  47. U-SQL Programming Improvements to Azure Data Lake Analytics for March 2016
  48. Saving Spark Distributed Data Frame (DDF) To PowerBI
  49. Extending Spark with Extension Methods in Scala: Fun with Implicits
  50. PySpark: Appending columns to DataFrame when DataFrame.withColumn cannot be used

2015 5

  1. Copy data easily from Azure Storage Blobs to Azure Data Lake Store
  2. Organize and discover your big data in the Azure Data Lake with Azure Data Catalog
  3. How To: Increase number of reducers in your Hive/MapReduce job
  4. How To: output file as a CSV using Hive in Azure HDInsight
  5. Hello world!