'%3e%3cpath%20d='M8%200C12.4183%200%2016%203.58172%2016%208C16%2012.4183%2012.4183%2016%208%2016C3.58172%2016%200%2012.4183%200%208C0%203.58172%203.58172%200%208%200ZM11.6162%204.38379C11.2257%203.99337%2010.5927%203.99338%2010.2021%204.38379L8%206.58594L5.79785%204.38379C5.40732%203.99334%204.77429%203.99329%204.38379%204.38379C3.99331%204.77429%203.99335%205.40733%204.38379%205.79785L6.58594%208L4.38379%2010.2021C3.99348%2010.5927%203.99341%2011.2257%204.38379%2011.6162C4.77426%2012.0066%205.40734%2012.0065%205.79785%2011.6162L8%209.41406L10.2021%2011.6162C10.5927%2012.0066%2011.2257%2012.0067%2011.6162%2011.6162C12.0067%2011.2257%2012.0066%2010.5927%2011.6162%2010.2021L9.41406%208L11.6162%205.79785C12.0066%205.40735%2012.0066%204.77429%2011.6162%204.38379Z'%20fill='%23080E17'%20fill-opacity='0.46'/%3e%3c/g%3e%3cdefs%3e%3cclipPath%20id='clip0_3761_713'%3e%3crect%20width='16'%20height='16'%20fill='white'/%3e%3c/clipPath%3e%3c/defs%3e%3c/svg%3e)
Fix Microsoft Fabric Dataflow Error When BigQuery Tables Are Hidden
Question details
The user is experiencing a dataflow failure when connecting Microsoft Fabric Gen1 to Google BigQuery.
- Product
- Microsoft Fabric
- Device & OS
- not provided
- Scenario
- Running or refreshing a dataflow that connects to Google BigQuery where some tables are hidden or have loading disabled.
- Observed behavior
- The dataflow fails and throws a Storage API 'Stream removed' error.
Ensure you have workspace administrator permissions in Microsoft Fabric and the necessary viewer roles in your Google BigQuery project to access refresh logs and table configurations.
Consult the Microsoft Fabric Community
Because this is a specific backend integration and Storage API issue, escalating it to Microsoft Fabric experts is the most effective way to find a workaround.
The 'Stream removed' error often indicates an unexpected interruption in the data stream between the BigQuery Storage API and Microsoft Fabric. When tables are hidden or have loading disabled, metadata mismatches can trigger this failure. Engaging with the specialized community helps Microsoft engineers track and address this specific bug.
Document your dataflow setup, focusing on the queries that connect to Google BigQuery and the specific settings applied to the hidden or disabled-loading tables.
Navigate to the refresh history of your dataflow and copy the complete error details, specifically the Storage API 'Stream removed' message.
Visit the official Microsoft Fabric Community support page and create a new post detailing your issue. Make sure to omit any sensitive business data or credentials from your screenshots and logs.
Analyze Your Exported BigQuery Data with WPS Office
While troubleshooting complex Microsoft Fabric dataflow connections, you may need to manually export and analyze your BigQuery data. WPS Office is a free, lightweight, and highly compatible alternative to Microsoft Office, making it easy to open large CSV exports and Excel files without expensive subscriptions.
- 1. Export data from BigQuery: Run your query in the Google Cloud Console and select the option to export the results as a CSV file to your local drive.
- 2. Open with WPS Spreadsheet: Launch WPS Office, select 'Spreadsheet', and open your downloaded CSV file.
- 3. Analyze your data: Use WPS Spreadsheet's built-in data tools to format, filter, and create pivot tables from your dataset.
Frequently Asked Questions
What does the 'Stream removed' error mean in Google BigQuery?
This error generally occurs when the BigQuery Storage API abruptly terminates a data stream. In the context of Fabric, it can happen if the metadata for hidden tables causes the connector to read invalid or unexpected schema states.
How do I disable load for a table in Microsoft Fabric dataflow?
Open your dataflow in Power Query Online, right-click the specific query or table on the left panel, and uncheck the 'Enable load' option. This keeps the query available for transformations but prevents it from loading into the final destination.
Where can I find the refresh history for my dataflow?
Go to your Microsoft Fabric workspace, locate your dataflow item, click the ellipsis (...) next to its name, and select 'Refresh history'. This will display a list of all successful and failed runs along with detailed error logs.
